The current (KDE 4.[3,4]) design of the SingleFileResource (SFR) is heavily based on the ical/vcard resources. Those two read the whole file at once and keep the entries in memory as long as they exists. However this is not really in line with the Akonadi design, as Akonadi itself keeps items in chache also and has advanced cache policies even. Also, other resources, like the mbox resource don't keep the whole file in memory but only store pointers to the beginning of each new entry. These two concepts conflict with each other as soon as Akonadi detects that the file on disk has changed. Currently SFR makes a backup of the file by calling writeFile() with a different file name. But this can only succeed correctly if the resource has all data in memory. This is not the case for the mbox resource (and this shouldn't be the case for the other SFR based resources in the future) which will result in a backup file that partly consist of the new data and partly the old data (if it succeeds at all).
Rational: Data caching is the responsibility of the Akonadi server, controlled by Cache policies. Permanently keeping data in resource memory wastes memory and is a bad example for developers interested in creating resources who might be looking for example code.
Rational: Akonadi calls this method when it needs items. If it doesn't call it, it doesn't need them. The need could be a client explicitly fetching the items or cache policy interval checking.
If it is deemed necessary to have all items loaded once a resource is configured, we can always just call synchronize() instead of synchronizeCollectionTree(), thus getting the retrieveItems() call.
Rational: a copy of the data file is needed to be able to detect/extract changes on item level when a file change notification happens. As a bonus handling of remote files does no longer need separate code paths.
Rational: does not require ItemSync, usually only very few changes, e.g. non-Akonadi application modifying it.
Rational: if no load has happend yet, we don't have to reload. If items have been loaded before, clearing the cache means the server will request them when necessary.
Make file reading/parsing ItemFetchJob like, i.e. create type specific file parse job and let it emit Items.
Simple implementation: emit items with full payload Advanced implementation: allow emitting of basic items, e.g. when processing the MBOX working copy which has offsets into the file and can read messages on demand
Process working copy and downloaded file in parallel. Have to hashes remoteId->item, one for each file.
For each item:
When both parsers are finished, re-process new item hash, this time also remove items from own hash if found in other. All items remaining in working copy hash have been removed in new file. All items remaining in new item hash, have been added.