Development/Tutorials/Akonadi/SerializerPlugin

Using your own data type with Akonadi

Tutorial Series	Akonadi Tutorial
Previous	C++, Qt, KDE development environment
What's Next
Further Reading	CMake, Akonadi Development Tools

Warning

This page needs a review and probably holds information that needs to be fixed.

Parts to be reviewed:

Port to KF5

This tutorial will guide you through the steps of getting your own data type ready for usage with Akonadi.

Akonadi itself has been designed to be able to handle any kind of data, so it needs to be told what kind of data an item is actually holding and how to convert it between programming language data structures and the universal array of bytes (sometimes also referred to as an octet stream).

Once you have support for the basic handling of your data type as an item payload, you can proceed to write a resource for persistant storage of the data. For details on doing that see the Akonadi Resource Tutorial

Prerequisites

Warning

This section needs improvements: Please help us to

cleanup confusing sections and fix sections which contain a todo

Describe required versions and build setup

Preparation

To use your own data type with KDE's API for Akonadi, you will need three things:

• a MIME type for identifying your data
• a programming language data structure (e.g. class) with value behavior
• methods or helper classes to write your data into a byte array and read from a byte array

MIME Type

Akonadi uses the MIME type of items to check whether one if its collections can be used as a target for new items carrying data of that MIME type. It is also used to map to a serializer plugin, i.e. a helper class which can create the high level data structures of your data type given a byte array and vice versa.

Depending on what your data type is there might be a standard MIME type, e.g. you are the first to implement Akonadi support for it, or you might already be using one for file associations, etc.

Otherwise you will have to "invent" one, e.g. by using the MIME "namespace" reserved for unofficial types, which start with application/x-. If your application is part of the KDE application suite, you might want to use a MIME type of the following form: application/x-vnd.kde.yourappname

Tip

This becomes important once you ship your Akonadi support. It is the identifier for your data type.

Data structure with value behavior

KDE's implementation of an Akonadi API, or more precisely the Akonadi::Item payload methods, need a high level representation of your data in the form of a class which's instances can be copied and assigned like values.

This can either be done by only having members which have value like behavior themselves, or by using a reference counting approach where instances share a pointer to the actual data.

The latter approach can most easily be achieved by using QSharedDataPointer and QSharedData together with the d-pointer idiom. As an example see how this is being used for the class which represents a KDE address book contact, KABC::Addressee

If a lot of your current API is pointer based and you'd rather not have to reimplement your data classes under the approach described above, you can work around this using an explicit shared pointer such as boost::shared_ptr. As an example see how this is being used for the class which represents a KDE calendar entry, KCal::Incidence: code using instances of this class with Akonadi create a typedef like this

#include <boost/shared_ptr.hpp>

typedef boost::shared_ptr<KCal::Incidence> IncidencePtr;

and then use this type with the item's payload methods.

Tip

Consider this only as a fallback. Having to remember pointer ownership through various levels of indirection and sharing will make debugging a lot more difficult.

Since your data classes will be used by your application, your Akonadi resource and by the serializer plugin, you will either have to build them as sub projects of a containing master project or put them into a shared library so they can be used by all three targets.

Data Serialization

Unless you are starting with a totally new application, you will already have means to save your data type to a file or something similar.

If you can abstract this to use QIODevice rather an QFile, or have already done so, you can use the same classes or methods to handle the data serialization for Akonadi.

Tip

Consider doing this anyway, since your items will have to be persistenly stored by an Akonadi resource anyway and the most simple form of persistant storage is a local file.

Tutorial Example

In order to demonstrate certain aspects of developing an Akonadi serializer plugin, consider the following example as a starting point similar to your situation.

The data type in this example is the history or chat log of an instant messaging client, i.e. a recording of the text messages between the local user and a single (for the sake of simplicity) remote user.

Data Format

The file format or rather serialization format is assumed to be a simple XML based one, basically looking like this:

<history>
  <header>
    <local id="contactId1" />
    <remote id="contactId2" />
  </header>
  <messages>
    <message who="contactId1" when="2009-01-01T12:00:00">Hello</message>
    <message who="contactId2" when="2009-01-01T12:01:10">Hi</message>
    <message who="contactId1" when="2009-01-01T12:02:04">Video at my place, 20:00?</message>
    <message who="contactId2" when="2009-01-01T12:04:00">Sure, I'll bring popcorn</message>
    <message who="contactId1" when="2009-01-01T12:04:50">Great! See you later</message>
    <message who="contactId2" when="2009-01-01T12:05:40">Yeah, cya!</message>
  </messages>
</history>

Data Classes

The classes for handling the history basically look like this:

class History
{
public:
  class Message
  {
  public:
    typedef QList<Message> List;

    Message();

    void setSender( const QString &contactId );

    QString sender() const;

    void setText( const QString &text );

    QString text() const;

    void setTimestamp( const QDateTime &timestamp );

    QDateTime timestamp() const;

  private:
    class Private;
    QSharedDataPointer<Private> d;
  };

  History();

  void setLocalContactId( const QString &contactId );

  QString localContactId() const;

  void setRemoteContactId( const QString &contactId );

  QString remoteContactId() const;

  void addMessage( const Message &message );

  Message::List messages() const;

  static QString mimeType();

private:
  class Private;
  QSharedDataPointer<Private> d;
};

The full version can be downloaded here: history.h, history.cpp

Both the main class as well as the nested class use the facilities provided by QSharedDataPointer and QSharedData to implement the value like behavior required by Akonadi::Item's payload methods.

Data I/O

The class for reading and writing the XML format look basically like this:

class HistoryXmlIo
{
public:
  static bool writeHistoryToXml( const History &history, QIODevice *device );

  static bool readHistoryFromXml( QIODevice *device, History &history );
};

The full version can be downloaded here: historyxmlio.h, historyxmlio.cpp

Getting started

All serializer plugins are implementations of the Akonadi::ItemSerializerPlugin interface.

We can kick-start the serializer plugin by using KAppTemplate, which can be found as KDE template generator in the development section of the K-menu, or by running kapptemplate in a terminal window.

Note

This template is only available with KDE SC 4.3 and later

First, we select the Akonadi Serializer Template in the C++ section of the program, give our project a name and continue through the following pages to complete the template creation.

A look at the generated project directory shows us the following files:

akonadi-serializer.png
akonadi_serializer_imhistory.desktop
CMakeLists.txt
README
akonadi_serializer_imhistory.cpp
akonadi_serializer_imhistory.h

At this stage it is already possible to compile the plugin, so we can already check if our development environment is setup correctly by creating the build directory and having CMake either generate Makefiles or a KDevelop project file.

Generating Makefiles

From within the generated source directory:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=debugfull ..

and run the build using make as usual.

Generating a KDevelop project file

From within the generated source directory:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=debugfull -G KDevelop3 ..

and open the generated project with KDevelop and run the build process from there.

Adjusting the plugin description file

The capabilities of the serializer plugin need to be described in both human understable and machine interpretable form. This is achieved through a so-called desktop file, similar to those installed by applications.

Since the akonadi_serializer_imhistory.desktop file generated by KAppTemplate contains only example values, we need to edit it:

[Misc]
Name=Instant Messaging History Serializer
Comment=An Akonadi serializer plugin for Instant Messaging History data

[Plugin]
Type=application/x-vnd.kde.imhistory
X-KDE-Library=akonadi_serializer_imhistory

Name and Comment strings might be visible to the user and can be translated. Since our plugin works on our instant messaging data, the MIME type field Type needs to set accordingly.

Adding the data specific classes to the build

As already mentioned in the Data Serialization section, the usual way to have the code needed for data processing shared between the application, the resource, and the serializer plugin is to put the respective classes into a shared library.

However, for the purpose of this tutorial we are just going to compile them into the plugin by adding them to the CMakeLists.txt file by changing

set( akonadi_serializer_imhistory_SRCS
  akonadi_serializer_imhistory.cpp
)

to this

set( akonadi_serializer_imhistory_SRCS
  akonadi_serializer_imhistory.cpp
  history.cpp
  historyxmlio.cpp
)

The files can be downloaded here: history.h, history.cpp, historyxmlio.h, historyxmlio.cpp

Full Payload

The most simple way of serialization is to always transfer the whole data between the in-memory data structure and the external format. This is usually also the method already implemented in data I/O classes, therefore the example XML I/O code implements this behavior as well.

First we need a couple of new include directives:

#include "history.h"
#include "historyxmlio.h"

#include <akonadi/item.h>

Then we implement the serialize method like this:

void SerializerPluginIMHistory::serialize( const Item& item, const QByteArray& label, QIODevice& data, int &version )
{
  Q_UNUSED( version );

  if ( label != Item::FullPayload || !item.hasPayload<History>() )
    return;

  const History history = item.payload<History>();

  HistoryXmlIo::writeHistoryToXml( history, &data );
}

and the deserialize method like this:

bool SerializerPluginIMHistory::deserialize( Item& item, const QByteArray& label, QIODevice& data, int version )
{
  Q_UNUSED( version );

  if ( label != Item::FullPayload )
    return false;

  History history;
  if ( !HistoryXmlIo::readHistoryFromXml( &data, history ) )
    return false;

  item.setPayload<History>( history );

  return true;
}

Given existing data I/O facilities, doing full payload serializations is rather trivial and totally sufficient for most data types.

Partial Serialization

In cases where it is possible to split the data into identifyable parts, serializing such parts independently can be used to reduce the amount of data to be transferred between Akonadi and its clients. For example only getting the data necessary to populate a list view and getting the rest once the user indicates interest in a specific item.

In the case of the example used in this tutorial, the header identifying the communication partners and the list of messages are such identifyable and separately transmittable parts.

Lets first implement this in the XML I/O helper class.

In file history.h we add to identifier contants:

static const char *HeaderPayload;
static const char* MessageListPayload;

and define them in file history.cpp:

const char *History::HeaderPayload = "Header";
const char *History::MessageListPayload = "MessageList";

In the file historyxmlio.h we add two method pairs for reading and writing the individual parts.

static bool writeHistoryHeaderToXml( const History &history, QIODevice *device );

static bool readHistoryHeaderFromXml( QIODevice *device, History &history );

static bool writeHistoryMessagesToXml( const History &history, QIODevice *device );

static bool readHistoryMessagesFromXml( QIODevice *device, History &history );

In file historyxmlio.cpp we implement them as follows:

bool HistoryXmlIo::writeHistoryHeaderToXml( const History &history, QIODevice *device )
{
  if ( device == 0 || !device->isWritable() )
    return false;

  QXmlStreamWriter writer( device );
  writer.setAutoFormatting( true );

  writer.writeStartDocument();

  writeHeader( history, writer );

  writer.writeEndDocument();

  return true;
}

bool HistoryXmlIo::readHistoryHeaderFromXml( QIODevice *device, History &history )
{
  if ( device == 0 || !device->isReadable() )
    return false;

  QXmlStreamReader reader( device );

  History partialHistory;
  if ( !readHeader( reader, partialHistory ) )
    return false;

  // only overwrite header fields
  history.setLocalContactId( partialHistory.localContactId() );
  history.setRemoteContactId( partialHistory.remoteContactId() );

  return true;
}

bool HistoryXmlIo::writeHistoryMessagesToXml( const History &history, QIODevice *device )
{
  if ( device == 0 || !device->isWritable() )
    return false;

  QXmlStreamWriter writer( device );
  writer.setAutoFormatting( true );

  writer.writeStartDocument();

  writeMessages( history, writer );

  writer.writeEndDocument();

  return true;
}

bool HistoryXmlIo::readHistoryMessagesFromXml( QIODevice *device, History &history )
{
  if ( device == 0 || !device->isReadable() )
    return false;

  QXmlStreamReader reader( device );

  History partialHistory;
  if ( !readMessages( reader, partialHistory ) )
    return false;

  // save header fields and then overwrite external instance
  partialHistory.setLocalContactId( history.localContactId() );
  partialHistory.setRemoteContactId( history.remoteContactId() );

  history = partialHistory;

  return true;
}

The plugin's serialize method changes to this:

void SerializerPluginIMHistory::serialize( const Item& item, const QByteArray& label, QIODevice& data, int &version )
{
  Q_UNUSED( version );

  if ( !item.hasPayload<History>() )
    return;

  const History history = item.payload<History>();

  if ( label == Item::FullPayload ) {
    HistoryXmlIo::writeHistoryToXml( history, &data );
  } else if ( label == History::HeaderPayload ) {
    HistoryXmlIo::writeHistoryHeaderToXml( history, &data );
  } else if ( label == History::MessageListPayload ) {
    HistoryXmlIo::writeHistoryMessagesToXml( history, &data );
  }
}

and the deserialize method to this:

bool SerializerPluginIMHistory::deserialize( Item& item, const QByteArray& label, QIODevice& data, int version )
{
  Q_UNUSED( version );

  // readHistoryHeaderFromXml() and readHistoryMessagesFromXml() only get parts of a full history item.
  // therefore make sure that the eventually already fetched other part is not lost by initializing
  // the local instance with the current payload if it exists.
  History history;
  if ( item.hasPayload<History>() )
    history = item.payload<History>();

  if ( label == Item::FullPayload ) {
    if ( !HistoryXmlIo::readHistoryFromXml( &data, history ) )
      return false;
  } else if ( label == History::HeaderPayload ) {
    if ( !HistoryXmlIo::readHistoryHeaderFromXml( &data, history ) )
      return false;
  } else if ( label == History::MessageListPayload ) {
    if ( !HistoryXmlIo::readHistoryMessagesFromXml( &data, history ) )
      return false;
  } else
    return false;

  item.setPayload<History>( history );

  return true;
}

and the parts method like this:

QSet<QByteArray> SerializerPluginIMHistory::parts( const Item &item ) const
{
  QSet<QByteArray> partIdentifiers;

  if ( item.hasPayload<History>() ) {
    const History history = item.payload<History>();

    bool hasHeader = false;
    bool hasMessageList = false;

    if ( !history.localContactId().isEmpty() && !history.remoteContactId().isEmpty() ) {
      partIdentifiers << History::HeaderPayload;
      hasHeader = true;
    }

    if ( !history.messages().isEmpty() ) {
      partIdentifiers << History::MessageListPayload;
      hasMessageList = true;
    }

    if ( hasHeader && hasMessageList )
      partIdentifiers << Item::FullPayload;
  }

  return partIdentifiers;
}

The two identifiers History::HeaderPayload and History::MessageListPayload can now be used with Akonadi::ItemFetchScope to only get the respective portion of the data.

For example in combination with an Akonadi::ItemModel to just show all available conversations without loading all their messages:

Akonadi::ItemModel *model = new Akonadi::ItemModel();
model->fetchScope().fetchPayloadPart( History::HeaderPayload );

Reference Implementation

The files used for the tutorials reference implementation can be found here: akonadi_serizalizer_imhistory.h and akonadi_serizalizer_imhistory.cpp