KDE PIM/Akonadi

From KDE TechBase

Akonadi TODO

The following list contains the things which need to be done for Akonadi.

Note: The person noted in the "Contact" column is not necessarily the one implementing that feature, but the one who can tell you what to do and how to help, i.e. you can also contribute to those tasks ;-)

Core

4.1

Urgent tasks that need to be finished for the KDE 4.1 release (May 19th):

Status Item Description Contact
DONE Item streaming in ItemSync/ResourceBase As discussed in Osnabrueck <Tom/Volker>
DONE Payload serialization format versioning see below

[email protected]

<Tobias>
DONE API for additional item parts As discussed in Osnabruck <Volker/Tobias>
DONE Infrastructure for showing additional dialogs from agents/resources As discussed in Osnabrueck <Tom>
DONE Allow to limit ItemFetchJob to current cache content Prevents search index feeder agents from downloading all remote data Volker <[email protected]>
DONE Fix API for item/collection modifications See Osnabrueck meeting notes for details Volker <[email protected]>
DONE Plugin Versioning For serializer plugins, also check using Qt plugin stuff instead of the libkdepim plugin framework Tobias <[email protected]>
DONE Tray app small app to control the server and have a something that can report errors Toma <[email protected]>
DONE Backup support Dump & Restore database contents, also useful when upgrading to a newer database version <[email protected]>

Post 4.1

Status Item Description Contact
IN PROGRESS Migration of kcal and kabc As bridge resources [mailto: <>]
TO DO Better error detection Sane error message when the server is not running or if there is some other problem communicating with it. This is a must-have for 4.2 when planning to migrate the resources. [mailto: <>]
IN PROGRESS Item size Needed by Mailody <[email protected]>
IN PROGRESS Finish collection model/view see below <[email protected]>
IN PROGRESS Collection statistics for sub-trees Provide a CollectionStatus object covering the full sub-tree in the model, allowing accumulated unread counts etc. <Thomas>
TO DO Batch jobs for modifying/deleting collections/items it would be great to have jobs which perform operations on several entities in one sitting (a use-case: I want to modify a set of items to reset the \Seen flag). [mailto: <>]
TO DO Batch job to retrieve a set of items from Akonadi Those items don't belong to the same collection, rather they are located in different collections [must have] [mailto: <>]
TO DO CollectionFetchJob/ItemFetchJob should be able to retrieve entities
  • by flags [must have]: I want to retrieve all items with the \Seen flag set.
  • by remote ids [must have]: I'm not sure whether that would improve the performance of ItemSync but at least I'll be able to implement a fast sync algorithm for RSS items

    This is problematic with change notifications, as they have to know about the filtering with the \Seen flag as well. This type of filtering would be possible with full Nepomuk interface though. We don't want yet another query language here. The plan is to fix the nepomuk agent and use that as semi-public interface for now.
    		• [mailto: <>]
    TO DO ResourceBase::collectionsRetrievalDone is missing I'm working around by calling collectionsRetrievedIncremental() with empty collection lists [mailto: <>]
    TO DO Provide a new command 'LINK' Akonadi resources should be able to link items to virtual/search collections [must have]. Since the RSS resource is going to be the first user of this feature, I can take a stab at this (after SoC).

    Probably possible for 4.2, important basis for the search stuff.

    		[mailto: <>]

    Post 4.1 or Later (Probably Post 4.2)

    Status Item Description Contact
    TO DO Error reporting Akonadi::Job basically has only one error code: Unknown <Tobias>
    IN PROGRESS Akonadi Artwork Icon for the tray app Toma <[email protected]>
    TO DO Nepomuk integration Generic Item tag/rate/comment, etc. <Tom>
    TO DO Account creation wizard as discussed in Berlin, see below [mailto: <>]
    TO DO Sync collection tree after creating setting up a resource see AgentInstanceCreateJob [mailto: <>]
    TO DO Support standard commands for QUndo framework [mailto: <>]
    TO DO Conflict detection in resources See Osnabrueck meeting notes for details [mailto: <>]
    IN PROGRESS Action framework see below Volker <[email protected]>
    TO DO Solid integration Switch online/offline state in ResourceBase automatically [mailto: <>]
    TO DO Make unittests work without destroying the production database [mailto: <>]
    IN PROGRESS Fix API docs The libakonadi move as well as the huge API changes following it broke lots of the docs technical and content-wise, see below <Tobias>
    TO DO Filesystem Backend Store content data in files instead of the database, transfer filehandles instead of data to the client [mailto: <>]
    TO DO Migration Data and settings from pre-Akonadi times, see below [mailto: <>]
    TO DO Remote Search Delegating searches to resources, see below [mailto: <>]
    TO DO Extended notifications Item change notification do not yet include their source collections (real and virtual) [mailto: <>]


    TO DO Parallel execution of jobs within an Akonadi::ResourceBase I'm using a custom D-Bus interface to start parallel jobs.

    This will get very complicated. Maybe use a proxy ResourceBase for this, which is thread-pooled? Probably not for 4.2, this gets complicated

    		[mailto: <>]


    TO DO Monitor::itemChanged() and Monitor::itemRemoved() don't say which collection the changed/removed item belongs to

    If the item resides in several collections (normal + virtual collections) then emit those signals for each collection. (must have). I am working around by keeping a mapping 'item -> its collection'

    Adding this would mean changing it in many places (Monitor, Observer, ...), this is lots of work but probably the right thing to do. Related to virtual collections, since with them, items can be in multiple collections. Better wait until the Nepomuk stuff is finished?

    		[mailto: <>]

    Supported Types

    Overview on the current state of support for various types:

    Type Serializer Multipart Support Models Views Resources Notes
    Email yes partial MessageModel, MessageCollectionModel, threading proxy model ? maildir, IMAP threading agent
    News yes (1) partial (1) (1) (1) NNTP
    Contact yes no ContactModel - vCard, facebook, KRes
    Events yes no EventModel - iCal, KRes
    Notes no no - - - not started yet
    Feeds yes no Feeds, Items ? OPML GSoC in playground/pim
    Bookmarks yes no - - local bookmarks, del.icio.us

    (1) see Email

    Mail specific extensions

    Status Item Description Contact
    TO DO Extend Message Model Add all required columns, save/restore layout, flag delegate, configurable columns, etc. [mailto: <>]
    TO DO Share mail flag code Standard actions, standard flag enum/constants, Nepomuk interaction [mailto: <>]
    TO DO Extend ENVELOPE Add more information to the envelope payload part in the serializer plugin, eg. trheading information is missing [mailto: <>]
    TO DO Fix/finish threading [mailto: <>]

    Event/Todo/Journal specific extensions

    Status Item Description Contact
    TO DO Todo proxy model See KOrganizers To-Do view <Bruno?>
    IN PROGRESS Subtype identification See discussion on kde-pim mailinglist <Kevin?>
    TO DO Agenda view KOrganizer agenda view based on Akonadi <Bruno?>
    TO DO Month view KOrganizer month view based on Akonadi <Bruno?>
    TO DO Timeline view KOrganizer timeline view based on Akonadi [mailto: <>]

    Resources, Agents and others

    Status Item Description Contact
    IN PROGRESS KResource Akonadi bridge for applications using KCal or KABC Kevin <[email protected]>
    IN PROGRESS Akonadi KResource bridge for data accessed through KCal or KABC resources Kevin <[email protected]>
    TO DO Expire Agent [mailto: <>]
    TO DO MBOX Resource <Thomas>
    TO DO Extend IMAP Resource [mailto: <>]
    TO DO POP3 Resource <Thomas>
    IN PROGRESS RSS Resource GSoC in playground/pim (Details) <Dmitry>
    TO DO Filter Agent [mailto: <>]
    TO DO Search [mailto: <>]
    TO DO Filter Rule GUI Used by filters and searches [mailto: <>]

    Resource status overview (this should list all resources existing in KDE3 or already under development for Akonadi):

    Resource Retrieve Collections Retrieve Items Change Collections Change Items Configuration Notes
    iCal yes (4) yes (1) (2) no yes (5) yes does not yet watch for file changes
    vCard yes (4) yes (1) (2) no ? yes does not yet watch for file changes
    maildir yes (1) (4) yes (1) (2) ? ? yes
    mbox no no no no no not started yet, code exists in KMail
    IMAP yes (1) (4)? yes (1) (2) no no no code exists in kio_imap4 and Mailody, support for extensions: quota, ACL, annotations missing, what about Kolab and Scalix?
    POP3 no no no no no not started yet,code exists in kio_pop3
    NNTP yes (4) yes (2) n/a n/a yes Needs support for local collection names and collection hierarchy
    Local Bookmarks ? ? ? ? (3) Code in akonadi/resources
    OpenChange ? ? ? ? ? Code in akonadi/resources
    Facebook ? ? ? ? ? Code in playground/pim
    del.icio.us ? ? ? yes no Code in playground/pim
    KABC no yes no yes yes
    KCal no yes no yes yes

    (1) only full sync supported currently, need optimization (2) does not yet honor cache policy (3) still relies on QSettings for configuration and/or doesn't provide configuration over D-Bus (4) does not yet provide correct access control settings (5) only adding new items, not changing existing ones

    Akonadi Braindump

    Ideas/notes etc. on various open issues in Akonadi.

    Akonadi Standard Actions

    Idea: Have something like KStandardAction for Akonadi that not only includes the representation of the action but also its state management and the actual operations. Like libakonadi that should be splitted into a generic, type-independent part and be extensible for type-specific actions. This will enable code sharing among many applications and guarantee consistent actions everywhere.

    State management: watch selection models of a collection and/or item model.

    Use KXMLGUI for context menus in standard views to allow easy extensibility with custom actions.

    Generic actions:

    • new collection [done]
    • new virtual collection
    • delete collection [done for single selection]
    • copy collection [done]
    • cut collection
    • paste collection [done]
    • synchronize collection [done for single selection]
    • synchronize resource
    • synchronize everything
    • show collection properties dialog [done]
    • delete item(s) [done]
    • copy item(s) [done]
    • cut item(s)
    • paste item(s) [done]
    • paste native data
    • tag item(s)
    • comment item
    • rate item(s)
    • configure resource
    • add resource? (would need a type parameter, etc.)
    • delete resource
    • manage local subscriptions [done]
    • move to submenu
    • copy to submenu

    The list is definitely long enough to make this worthwhile.

    Collection Model / Collection View

    Ideas/missing features/bugs of the collection model/view:

    • Enable/disable status columns
    • Show status after the name (see KMail)
    • Size column
    • Save/restore layout
    • Custom collection icons (see KMail, also needed for resource defined special folders (eg. Inbox)
    • Status tooltips (see KMail in 3.5.9)
    • Quick search
    • Favorite folder view as proxy model on top of the normal collection model (FlatCollectionProxyModel might be helpful for there)
    • Accumulated statistics (unread, total, size)
    • Fix dnd: move/copy/cancel menu always shows up and never has any effect

    Compatibility

    Notes on how to keep long-term protocol, source and binary compatibility.

    • Detect server version in Akonadi::Session, might be useful in case of protocol extensions/changes
    • What about database server version updates?
    • Versioning or any other kind of serialization format meta data, we'll need that in case of changes in serialization formats (see eg. Robert's compression patch where this is needed)
    • We currently use 32 bit ids, probably hard to change later on, is that enough?
    • Plugin versioning

    Resource API issues

    Notes/ideas/complaints about the Akonadi::ResourceBase API:

    • Extremely error prone:
      • Scheduler dead-locks when a requested operation is not correctly announced as finished, esp. a problem in error cases.
      • Using the result methods multiple times or when not requested asserts
    • Item streaming is missing, requiring all data to be in memory at once
    • Non-incremental updates need to know the results of ItemSync to not have to provide all data, even for already existing/unchanged items.

    API / BC issues

    Smaller stuff that should be fixed before the ABI freeze and is not yet listed above:

    • Cleanup the D-Bus format used by NotificationMessage
    • No mentioning of "IMAP" in the public API, we are not using IMAP
    • Naming and installed location of libraries, headers and executables (see discussion on kde-pim ML)
    • Payload part labels are QStrings, attribute types are QByteArray

    Deployment issues

    • Multiple access: Should multiple Akonadi instances' mysqlds access a single set of data files the mysql will likely corrupt the data. This can happen in any NFS+YP installation where users can log onto any machine and access shared homes. MySQL relies on filesystem locking to prevent multiple access. MySQL multiple instance docu.
    • InnoDB tables should not be used on NFS.
    • NFS speed: MySQL documentation recommends against locating its data files on network shares.
    • AppArmor: Distros' AppArmor profiles prevent MySQL from writing outside its defined data directory (usually /var/lib/mysql). This is a problem at least with *buntu and openSUSE. These will need to be adapted. It is possible to daisy-chain profiles so that MySQL started by Akonadi receives a different profile to MySQL running standalone. An empty profile has all rights. Another possibility is to adapt the general MySQL profile so it can write to ${XDG_DATA_HOME}/akonadi(usually ${HOME/}.local/share/akonadi). Both support for profile chaining and ${HOME} may depend on the version of AppArmor installed. What about SELinux?
    • Backup: MySQL data files should not be backed up without telling the mysqld process, otherwise a corrupt backup will be made. LOCK TABLES and FLUSH TABLES at the least. A dump can be made or mysqlhotbackup may be used in some circumstances. We should consider sysadmins' backup techniques when planning/promoting Akonadi, as a simple rsync cronjob with running Akonadi will not work. MySQL backup advice.

    Account Creation Wizard

    Notes on the resource configuration wizard discussion.

    • WizardBase class
    • Wizard discovery via .desktop files
    • A wizard can configure multiple resources (eg. IMAP + LDAP for Kolab) and non-resources (mailtransport)
    • Optionally hide resources completely covered by wizards in the Add Account dialog
    • List wizards in the Add Account dialog
    • Store Account -> Resources information to offer deletion of depending resources

    PostgreSQL datastore support

    First review of the akonadi code reveals PGSQL Qt driver is already supported, but the code is not totally ready for it. Some first reading comments about where to patch for supporting PostgreSQL too:

    akonadi/server/src/storage/dbinitializer.cpp

    • line 201..., ALTER TABLE ADD [UNIQUE] INDEX syntax is MySQL only, better use the common one CREATE [UNIQUE] INDEX ... one [done with revision 789899]
    • line 303, QString DbInitializer::sqlType(const QString & type), LONGBLOB is the MySQL type, PostgreSQL will use BYTEA.
    • line 362, qFatal("Implement index support for your database!");
     mysql: SHOW INDEXES FROM %1
 pgsql: SELECT c2.relname, i.indisprimary, i.indisunique, i.indisclustered, i.indisvalid, pg_catalog.pg_get_indexdef(i.indexrelid, 0, true), c2.reltablespace
 FROM pg_catalog.pg_class c, pg_catalog.pg_class c2, pg_catalog.pg_index i
 WHERE c.oid = '16713' AND c.oid = i.indrelid AND i.indexrelid = c2.oid
 ORDER BY i.indisprimary DESC, i.indisunique DESC, c2.relname
 
    relname    | indisprimary | indisunique | indisclustered | indisvalid |                       pg_get_indexdef                        | reltablespace
 --------------+--------------+-------------+----------------+------------+--------------------------------------------------------------+---------------
  foo_pkey     | t            | t           | f              | t          | CREATE UNIQUE INDEX foo_pkey ON foo USING btree (id)         |             0
  foo_time_key | f            | t           | f              | t          | CREATE UNIQUE INDEX foo_time_key ON foo USING btree ("time") |             0
 (2 lignes)

    akonadi/server/src/storage/datastore.cpp

    • line 700, qint64 DataStore::highestPimItemId() const, seems wrong for a concurrent program (SELECT MAX(col) FROM table; is not reliable if another thread is inserting at the same time) : check the callers
    • line 794, qint64 DataStore::lastInsertId( const QSqlQuery & query ), PostgreSQL doesn't provide concurrent-bogus last_insert_id, but from 8.2 accepts the syntax INSERT INTO ... RETURNING col1, col2, ...;
    • line 806, QString DataStore::dateTimeFromQDateTime( const QDateTime & dateTime ), use PostgreSQL to_char() formatting function for date rather than having Qt do it

    Documentation

    • Avoid examples using KJob::exec() due to its problematic side-effects.
    • Examples in the ItemCreateJob docs (and maybe others) operate on items in the root collection which does not work at all
    • Resurrect the server API docs

    Migration of pre-Akonadi data and settings

    • KCal/KABC
      • switch KRes resources to Akonadi KRes compat resources, use KRes Akonadi compat resource, basically injecting Akonadi inbetween
      • Replace Akonadi KRes compat resources with their ported equivalent
      • Application side does not need migration code, "only" port to Akonadi
    • KMail
      • Should migrate to one resource per account + one for previous local folders
      • Problem: local folders are mixed mbox/maildir
        • Dedicated KMail compat resource?
        • Convert everything to maildir first?
      • Flags are stored in proprietary binary format
    • KNode
      • Migrate every account to NNTP resource
      • Local flags should be preserved, stored in proprietary binary format
      • Local subscription state should be preserved (where is that stored?)
      • Local folders using MBOX format + proprietary binary index
    • Akregator

    Search

    • Delegation to resources still completely missing
      • Requires query transformation
      • Requires management for live searches
      • Requires the ability to report search results (needs protocol extension)
    • Query language still undecided, current options: XESAM, SPARQL
    • Backend still undecided, current options: XESAM, Nepomuk

    KMail Breakdown Plan

    The current plan is to put some parts of KMail into a stand-alone library, independent of KMail. This increases code reuse (for example, the message composer could be shared with Mailody) and makes the code a lot easier to maintain and to port to Akonadi.

    Status Item Description Contact
    TO DO Bodypart formatters [mailto: <>]
    TO DO Reader Window [mailto: <>]
    TO DO Composer: Editor [mailto: <>]
    TO DO Composer: Message Composer [mailto: <>]
    TO DO Composer: GUI Window [mailto: <>]
    TO DO Queue Manager for mailtransport [mailto: <>]
    TO DO Templates: Core [mailto: <>]
    TO DO Templates: GUI [mailto: <>]
    TO DO Port KMCommands [mailto: <>]
    TO DO Port away from KMMessage and KMFolder* everywhere it is left An idea might be: wrap KMMsgBase/KMMessage inside a reference counted Message class. Insulate everything else from it by wrapper functions. KMFolder should be easier... [mailto: <>]
    TO DO Migration application for index and other config [mailto: <>]
    TO DO Port MDNs [mailto: <>]


    The Road to World Domination

    Sort of related so the stuff above but with a scope for all of KDE PIM and beyond.

    Status Item Description Contact
    TO DO Share identity config GUI parts eg. the signature configuration widget [mailto: <>]
    TO DO Mailtransport Agent Global shared outbox, central mail sending instance [mailto: <>]
    TO DO Recently sent to support in mailtransport related to above, based on Akonadi/Nepomuk [mailto: <>]
    TO DO Composer engine Shared stuff for KMail/KNode/Mailody: Crypto, editor, editor actions, attachment model, message assembly, etc. [mailto: <>]

    Akonadi FAQ

    Where do I find the Akonadi config files?

    ~/.config/akonadi/

    Where does Akonadi store my data?

    ~/.local/share/akonadi/

    Which DBMS does Akonadi use?

    So far only MySQL. There is some work on PostgreSQL support going on though. Basically, every database that is supported by QtSQL can be used, requiring minimal changes in the code at most. However, not all of them provide the features needed by Akonadi (see next two questions).

    Why not use sqlite?

    We tried. Really. It can't handle the concurrent access very well, in the best case this means very slow operations, but we've also seen deadlocks and failing transactions. Once that's fixed in sqlite, adjusting Akonadi to use it again instead of MySQL is no problem.

    Why not use MySQL/Embedded?

    We tried that as well, there are two reasons for not using it: No support for the InnoDB engine (which we need for transaction support) and poor availability (only OpenSUSE provided usable packages, needed a patched QSQL driver).

    Do I need a running MySQL server?

    No. Akonadi starts its own local MySQL server (unless configured otherwise, see next question). All you need is having the 'mysqld' binary installed at runtime (usually found in the mysql-server package of your distribution).

    Can Akonadi use a normal MySQL server running on my system?

    Yes, it can. You find the corresponding settings in ~/.config/akonadi/akonadiserverrc.

    I don't like a database server because of backups/running on NFS/etc.

    See section "Deployment Issues" above, we are aware of that and working on it. Some of these, like backup/restore are already implemented. But please be aware that most of this issues also existed before (eg. with KMail's custom binary indexes).

    Can a single Akonadi instance be used by multiple users?

    No. There has to be one Akonadi server instance per user. However, it is possible to use a shared database server.

    Can I access the Akonadi server on a remote machine?

    No. Akonadi is not a groupware server. It's a local cache only.

    I have an unknown error 255 when running akonadictl on Ubuntu

    Distributions using Apparmor require you to run aa-complain mysqld with root privileges then reload apparmor.

    What's the difference to EDS?

    EDS is limited to contacts and calendar data, Akonadi is type-independent. Especially, Akonadi is designed to also handle high-volume data such as email. Akonadi and EDS also differ largely in their access protocol on the technical level (Corba/D-Bus vs. local socket with IMAP-like protocol/D-Bus) as well as one the protocol level (type specific vs. generic).

    Retrieved from "https://techbase.kde.org/index.php?title=KDE_PIM/Akonadi&oldid=31724"