User:Mark Gaiser/KDirchainRebuild

Introduction

This all started a few years ago when i was profiling Dolphin. I wasn't happy with it's general performance and was even less happy when it came to massive folders. Added to that was the - back then - brand new stuff called "QML". So i wanted to create a file browser that is fast, memory efficient and with a GUI done in QML so that i could add those nifty animations. So i set out to restructure all of the directory handling from KIO from as low UDSEntry till the user facing models. I named that "KDirchainRebuild" since it was rebuilding the chain of events when you open a directory.

That's how it all started a few years ago. Fast forward to today and you see an entirely different landscape. Dolphin is now very fast even on large folders. I didn't make any patches for it, but my initial research in why it was performing poorly was apparently a spark for the Dolphin devs to take a good close look at it and improve it greatly. The biggest thing i did here was investigating how entries where being batched in KIO. That is improved to do real batching which in turn allows an application to show data more quickly to the user. A big thank you here for David Faure since the new batching triggered another weird socket issue that he solved.

So is there still a reason to even consider this "KDirchainRebuild" over the existing classes? Well, yes. The new classes are aimed at simplicity, small memory footprint and being as fast as possible (which sometimes conflicts with simplicity), and - via models - being usable in QML.

Why new classes instead of improving the existing ones?

Initially the idea was to take the current classes, remove all function content and re-implement them in my view. Later i abandoned that idea because there seemed to be quite a few functions that i wanted to handle in a different way along with removing methods altogether. I wasn't really interested in marking a dozen functions as "deprecated". Specially not as long as this project was highly experimental and just for my own pet project. So the choice was easiest to just go for my own class names without the burden of having to re-implement the existing classes.

The Classes

Before i present the new classes and the API they have, i first want to take a moment to say that the classes are not done yet! The API is far from finished and code-wise it already can use a bit of cleanup. All the information below is just to indicate it's current state, what is still planned for it and where i see potential issues (and resolutions).

KDirectoryEntry

Source:


KDirectoryEntry compares mostly to KFileItem only with a greatly reduced memory footprint (and a lot of missing functions).

Data stored

Inherits from no other classes.

  • 1 d pointer (KDirectoryEntryPrivate)
  • 1 KIO::UDSEntry
  • 1 bool to know if details are loaded (details 0 vs 2). I still like to get rid of this one.

Ideas

You always need an UDSEntry if you want to know something about a file. It might be beneficial for the UDSEntry itself to know (with a bool) if it has all the file details or not. It adds a bool value, but that can be used later on to prevent needless hash lookups if you have no details loaded. I might be worth adding this or at least experiment with it.

For the new classes. I just like to have details with true or false (bool) instead of a string (0 or 2). Easy to implement on my side, but will it cause any other difficulties later on? I don't know.

Functions

Some of the below documentation is a direct copy from KFileItem.

   /**
    * Returns the raw file name with extension as it comes from UDSEntry.
    * @return QString
    */
   const QString name() const;
   /**
    * Returns the owner of the file.
    * @return QString
    */
   const QString user() const;
   /**
    * Returns the group of the file.
    * @return QString
    */
   const QString group() const;
   /**
    * Returns the base name of a file. This is the name without extension.
    * @return QString
    */
   const QString basename() const;
   /**
    * Returns the extension of a file without leading dot.
    * @return QString
    */
   const QString extension() const;
   /**
    * Returns the icon name based on the output of QMimeType
    * @return QString
    */
   const QString iconName() const;
   /**
    * Returns the icon comment based on the output of QMimeType
    * @return QString
    */
   const QString mimeComment() const;
   /**
    * Returns the file size from UDSEntry if details where loaded. 0 for no details or if the current entry is a folder.
    * @return QString
    */
   const KIO::filesize_t size() const;
   /**
    * This is used internally when constructing this object with an UDSEntry, or used externally to update this object. For example when a rename happens. Users using this class cannot use this object to get notified of changes. The KDirectory object containing this instance will notify you of changes.
    */
   virtual void setUDSEntry(const KIO::UDSEntry& entry, const QString& details = "0");
   /**
    * Returns true if this item represents a link in the UNIX sense of
    * a link.
    * @return true if the file is a link
    */
   bool isLink() const;
   /**
    * Returns true if this item represents a directory.
    * @return true if the item is a directory
    */
   bool isDir() const;
   /**
    * Returns true if this item represents a file (and not a a directory)
    * @return true if the item is a file
    */
   bool isFile() const;
   /**
    * Checks whether the file or directory is readable. In some cases
    * (remote files), we may return true even though it can't be read.
    * @return true if the file can be read - more precisely,
    *         false if we know for sure it can't
    */
   // STUB! To be implemented
   bool isReadable() const;
   /**
    * Checks whether the file or directory is writable. In some cases
    * (remote files), we may return true even though it can't be written to.
    * @return true if the file or directory can be written to - more precisely,
    *         false if we know for sure it can't
    */
   // STUB! To be implemented
   bool isWritable() const;
   // STUB! To be implemented
   bool isExecutable() const;
   // STUB! To be implemented
   bool isModified() const;
   // STUB! To be implemented
   bool isSystem() const;
   /**
    * Requests the modification, access or creation time, depending on @p which.
    * @param which the timestamp
    * @return the time asked for, QDateTime() if not available
    * @see FileTimes
    */
   QDateTime time(FileTimes which) const;
   /**
    * Checks whether the file is hidden.
    * @return true if the file is hidden.
    */
   bool isHidden() const;
   /**
    * Checks whether the file details are loaded.
    * @return true if details are loaded, false otherwise.
    */
   bool detailsLoaded() const;

KDirectory

Source:


You can't compare KDirectory to anything currently living in KDE (that i know of). It is somewhat inspired by QDir but even there not fully comparable.

KDirectory is basically a data management class. If you open a folder, this class will do a call to KIO::listDir and retrieve it's output. What you as a user can do is set properties how the output should be accessed. You can set filters and flags just like QDir. This class then applies those filters and flags to all incoming data and stores it (internally) in two lists. Once for those that pass, one for those that don't. No data is duplicated there. The number of items in both lists combined is the number of items you would see when you execute "ls -l | wc -l" in the same folder on the command line. This class only gives you access to the items that passed the filters/flags. If you want access to the items that didn't pass then you have to change your filters/flags to make them pass. Or set no filters/flags which makes all items pass. For bookkeeping, this class stores quite a bit of additional data.

Ideas

KDirectory should also manage file/folder changes in it's directory. However, the signals emitted by KDirWatch and QFileSystemWatcher are too generic. I need more detailed signals to let it be useful. Right now adding a file to a directlry simply means re-indexing the entire directory because we don't have enough details in our signals. Details that inotify (the backend) does provide. I plan to revive some old patches for QFileSystemWatcher on the frameworks sprint and make it work again. Once that is in working order, KDirectory can use them. Right now KDirectory just doesn't use any mechanism to monitor a folder for changes. Remember, work in progress!

Functions

   /**
    * Returns all entries that passed the filters and flags.
    * @return QVector<KDirectoryEntry>
    */
   virtual const QVector<KDirectoryEntry>& entries();
   /**
    * Entry returns the KDirectoryEntry object if it's index is in the filteredEntries.
    * @param index
    * @return KDirectoryEntry at index
    */
   virtual const KDirectoryEntry& entry(int index);
   /**
    * String of the full path for this directory.
    * @return QString
    */
   virtual const QString& url();
   /**
    * Number of entries that passed the filters and flags
    * @return int
    */
   virtual int count();
   /**
    * Set the details for the current directory. It can be 0 (no details) or 2 (full details). If the value passed is different then the current details value (and valid) then the directory will be re-scanned according to the new details value.
    * @param QString details. Either 0 or 2.
    * @return KDirectoryEntry at index
    */
   virtual void setDetails(const QString& details);
   // For those, see QDir documentation.
   QDir::Filters filter();
   void setFilter(QDir::Filters filters);
   QDir::SortFlags sorting();
   void setSorting(QDir::SortFlags sort);
   /**
    * Loads the entry details and passes it to the KDirectoryEntry that needs the information.
    * Be aware that this function is executing a (slow) stat call!
    * @param int id the id of the file to load. This is the QVector index id.
    */
   void loadEntryDetails(int id);

Signals

   /**
    * New entries in this folder have been processed.
    * @param KDirectory* directory pointer to the current directory. This pointer is given
    *        because you're likely to use multiple KDirectory objects so you wouldn't easily
    *        know which KDirectory object spawned this signal.
    */
   void entriesProcessed(KDirectory* dir);
   /**
    * Done loading entries.
    * @param KDirectory* directory pointer to the current directory. This pointer is given
    *        because you're likely to use multiple KDirectory objects so you wouldn't easily
    *        know which KDirectory object spawned this signal.
    */
   void completed(KDirectory* dir);
   /**
    * Done loading entries.
    * @param KDirectory* directory pointer to the current directory. This pointer is given
    *        because you're likely to use multiple KDirectory objects so you wouldn't easily
    *        know which KDirectory object spawned this signal.
    * @param int id is the id that you can use to get the KDirectoryEntry object (entry(id)).
    */
   void entryDetailsChanged(KDirectory* dir, int id);

KDirListerV2

Source:


KDirListerV2 had the intention to be comparable with KDirLister, but when i decided to move away from the current KDE class structure and go for my own versions i kept the name (KDirListerV2). Right now it can't be compared to KDirLister. The name of this class is also confusion now since it's not actually listing. It's just a store from url (QString) to KDirectory mapping. KDirectory is then executing all the difficult parts. This class still has value because it maintains (and keeps) the url's that have been passed through it. It's acting as a cache for those and returns them when requested. Much like KDirLister and KDirListerCache work.

I guess this class can be renamed to something more appropriate like KDirectoryLoader.

Data stored

Just one. Although the private class KDirListerV2Private is also having a pointer back to KDirListerV2. This is old cruft from earlier versions. I still need to remove it.

  • QHash<QString, KDirectory*> m_cache;

Ideas

Rename to KDirectoryLoader or some other name.

Perhaps some kind of LRU caching mechanism should be used. I did try this before, but was very annoying during development so i removed it. I was using the ALT-LRU classes. Perhaps good for later re-inclusion.

Regarding the openUrl function. I think i'd like to get rid of the flags parameter (and the OpenUrlFlags enum). The only real use for that was opening a folder or reloading one. Opening should be done through this route and reloading should be done directly on the KDirectory object representing the url (KDirectory::reload(), which doesn't exist yet). The Keep flag would be lost. Not a big loss imho. Right now reloading is done through KDirListerV2 which first removes the KDirectory object before creating a new one for a "reload". Seems a bit wasteful.

DirectoryFetchDetails

Just for reference since this will give some sense to the openUrl where this object is passed in.

   struct DirectoryFetchDetails {
       QString url;
       QString details = "0";
       QDir::Filters filters = QDir::NoFilter;
       QDir::SortFlags sorting = QDir::NoSort;
       KDirListerV2::OpenUrlFlags openFlags = OpenUrlFlag::NoFlags;
   };

Functions

   /**
    * Open a new URL to be listed.
    *
    * @param url the directory URL.
    * @param flags. Can be NoFlags, Keep or Reload
    * @return true if successful, false otherwise.
    */
   virtual bool openUrl(const QString& url, OpenUrlFlags flags = NoFlags);
   /**
    * Open a new URL to be listed.
    *
    * @param dirFetchDetails
    * @return true if successful, false otherwise.
    * @see DirectoryFetchDetails
    */
   virtual bool openUrl(DirectoryFetchDetails dirFetchDetails);
   /**
    * Test if a given URL is being listed.
    *
    * @return true if directory is listed, false otherwise
    */
   virtual bool isListing(const QString& url);
   *** I'm thinking of removing this function. It doesn't seem to make much sense.
   /**
    * Return a KDirectory pointer representing the given URL.
    *
    * @param url
    * @return KDirectory pointer if url is known, nullptr otherwise.
    */
   virtual KDirectory* directory(const QString& url);

Signals

   /**
    * NOTE: pay close attention here, This signal is returning the internal
    * KDirectory object which has ALL entries inside up till the emit moment!
    * It does not emit incremental updates!
    *
    * Also, there is no guarantee that this pointer will be the same for the
    * same folder. You should therefore not rely on this pointer.
    *
    * @param items
    */
   void directoryContentChanged(KDirectory* directoryContent);
   /**
    * @brief completed
    * @param items
    */
   void completed(KDirectory* directoryContent);

DirListModel

Source:


This is a flat model inheriting from QAbstractListModel. It exposes nearly all functions from a KDirectoryEntry class along with some extra stuff. Like a pixmap for previews. Although the pixmap support is currently broken :)

All roles are also exported by re-implementing roleNames. This means that QML can use this model without hassle. This model should be enough for exposing a filesystem to QML and using QML to graphically represent it.

FlatDirGroupedSortModel

Source:


This model is a QAbstractProxyModel implementation where a DirListModel is it's source. The intention of this model:

  1. [done] Sorting entries
  2. [done] Grouping entries
  3. [done] Sorting each individual group
  4. [started, unstable] Sorting only the area you see (on demand sorting)
  5. [nothing yet] Filtering based on input or rules (hidden files, only exe, only... etc...)


The first 3 are implemented. I can even demonstrate them in Accretion.

This is all done in one model because of QML. I initially had the same stuff working using nested models, but doing that also means nested views which was becoming really painful in QML. QML then can't use it's ListView optimizations when you nest them so you would end up with a gigantic content size within a listview that just didn't scale. Hence the need of a "flat" model. The actual missing thing here is QML lacking a treeview.

Result after discussion

At the frameworks sprint we (David Faure and i) have had a long discussion about some of the ideas in here compared to the classes currently in KIO.

The route we'd like to go for now is a mixture of my classes and the current KIO classes.

KFileItem

KFileItem currently is a big class and heavy to load. It might be more efficient (memory and cpu wise) to store all the information that KFileItem needs in the UDSEntry object. That's where the UDSEntry is for anyway so we're just going to use it more.

KDirLister

Currently in KDE there is no concept of "KDirectory". It's basically a big list of entries in KDirLister(Cache) which means that KDirLister has all directory specific stuff. My KDirectory is pretty much working already without the use of a "KDirLister(v2)" because KDirectory is handing the listing itself. A feature that KDirectory misses, but KDirLister has is tree based support (up and down). The idea now is to mix the two:

  • Get rid of KDirLister(2)
  • Give KDirectory tree support
  • Publicly you would just be opening a KDirectory object to get it's files. No more need for KDirLister.
  • KDirectory needs to maintain it's own (LRU) cache or some mechanism to prevent a memory explosion.

This page was last edited on 26 April 2014, at 14:06. Content is available under Creative Commons License SA 4.0 unless otherwise noted.