Projects/PIM/Development/SingleFileResourceRefactoring - Revision history

Krake: Implementation Ideas

2009-11-06T20:00:14Z

Implementation Ideas

Krake: /* Advantages */

2009-11-06T19:33:35Z

‎Advantages

Krake at 19:29, 6 November 2009

2009-11-06T19:29:35Z

Krake: Rationals, additional option

2009-11-06T19:27:16Z

Rationals, additional option

Krake: /* On item retrieval */

2009-11-06T19:17:54Z

‎On item retrieval

Krake: Add rationals

2009-11-06T19:16:11Z

Add rationals

Krake: Add rationals

2009-11-06T19:09:13Z

Add rationals

Bbroeksema: /* SingleFileResource Refactoring */

2009-11-06T17:30:11Z

‎SingleFileResource Refactoring

Bbroeksema: moved Projects/PIM/Development/Pim/SingleFileResourceRefactoring to Projects/PIM/Development/SingleFileResourceRefactoring

2009-11-06T17:09:45Z

moved Projects/PIM/Development/Pim/SingleFileResourceRefactoring to Projects/PIM/Development/SingleFileResourceRefactoring

Bbroeksema: Created page with '= SingleFileResource Refactoring ='

2009-11-06T17:08:22Z

Created page with '= SingleFileResource Refactoring ='

New page

= SingleFileResource Refactoring =

← Older revision		Revision as of 20:00, 6 November 2009
Line 52:		Line 52:
	* Changes which have not been written to the file yet only result in a conflict if the respective item is either changed between working copy and downloaded file or no longer present in downloaded file		* Changes which have not been written to the file yet only result in a conflict if the respective item is either changed between working copy and downloaded file or no longer present in downloaded file
	* Better data consistency in case files are changed by non-Akonadi programs		* Better data consistency in case files are changed by non-Akonadi programs
		+
		+	== Implementation Ideas ==
		+
		+	=== Asynchronous Parsing ===
		+
		+	Make file reading/parsing ItemFetchJob like, i.e. create type specific file parse job and let it emit Items.
		+
		+	Simple implementation: emit items with full payload
		+	Advanced implementation: allow emitting of basic items, e.g. when processing the MBOX working copy which has offsets into the file and can read messages on demand
		+
		+	=== Parallel Parsing ===
		+
		+	Process working copy and downloaded file in parallel.
		+	Have to hashes remoteId->item, one for each file.
		+
		+	For each item:
		+	* Check for respective item in other hash
		+	** On miss, put into own hash
		+	** On match check for change
		+	*** Report to Akonadi
		+	*** Remove from other hash
		+
		+	When both parsers are finished, re-process new item hash, this time also remove items from own hash if found in other.
		+	All items remaining in working copy hash have been removed in new file. All items remaining in new item hash, have been added.

@@ Line 50: / Line 50: @@
 === Advantages ===
 * Reduced memory usage
-* if dirty items are not part of the change, no need to backup or panic ???
+* Changes which have not been written to the file yet only result in a conflict if the respective item is either changed between working copy and downloaded file or no longer present in downloaded file
-* Better data consistency in case files are changed by non-Akonadi programs.
+* Better data consistency in case files are changed by non-Akonadi programs

@@ Line 45: / Line 45: @@
 === Drawbacks ===
-* Two files are parsed on fileChanged()
+* Potentially necessay to parse two files
 === Advantages ===

@@ Line 33: / Line 33: @@
 * read both copies, compare item by item, using any dirty item in favor of the respective one from copy 1.
 * do not call synchronize, modify Akonadi like any resource with item level change notification.
 * finally remove copy 1 and make copy 2 the new copy 1
 === Drawbacks ===

@@ Line 24: / Line 24: @@
 Rational: a copy of the data file is needed to be able to detect/extract changes on item level when a file change notification happens. As a bonus handling of remote files does no longer need separate code paths.
-* Create items from the copied file
+* Create items from the downloaded file
-* Compare with items from current working copy.
+* Compare with items from current working copy
 * Call itemsRetrieved
 === On file change ===

@@ Line 20: / Line 20: @@
 === On item retrieval ===
-* Retrieve item first makes a copy of the file (i.e. as done now for remote files but, now its done always)
+* Make loading always download the file independent of whether it is remote or local
 * Create items from the copied file
 * Call itemsRetrieved

@@ Line 8: / Line 8: @@
 === General ===
-* The file parsers should never store the complete data in memory
+* The file resources should never store the complete data in memory
-* When items are requested from the file parsers it always should read from the file it is initialized for.
+* When items are requested from the file resource it always should read from the file it is initialized for
-* The SFR only reads the file first at retrieveItems.
 === On item retrieval ===

← Older revision		Revision as of 17:30, 6 November 2009
Line 1:		Line 1:
	= SingleFileResource Refactoring =		= SingleFileResource Refactoring =
		+
		+	== Problem Statement ==
		+
		+	The current (KDE 4.[3,4]) design of the SingleFileResource (SFR) is heavily based on the ical/vcard resources. Those two read the whole file at once and keep the entries in memory as long as they exists. However this is not really in line with the Akonadi design, as Akonadi itself keeps items in chache also and has advanced cache policies even. Also, other resources, like the mbox resource don't keep the whole file in memory but only store pointers to the beginning of each new entry. These two concepts conflict with each other as soon as Akonadi detects that the file on disk has changed. Currently SFR makes a backup of the file by calling writeFile() with a different file name. But this can only succeed correctly if the resource has all data in memory. This is not the case for the mbox resource (and this shouldn't be the case for the other SFR based resources in the future) which will result in a backup file that partly consist of the new data and partly the old data (if it succeeds at all).
		+
		+	== Suggested Changes ==
		+
		+	=== General ===
		+	* The file parsers should never store the complete data in memory
		+	* When items are requested from the file parsers it always should read from the file it is initialized for.
		+	* The SFR only reads the file first at retrieveItems.
		+
		+	=== On item retrieval ===
		+	* Retrieve item first makes a copy of the file (i.e. as done now for remote files but, now its done always)
		+	* Create items from the copied file
		+	* Call itemsRetrieved
		+
		+	=== On file change ===
		+	* A second copy of the file is created.
		+	* read both copies, compare item by item, using any dirty item in favor of the respective one from copy 1.
		+	* do not call synchronize, modify Akonadi like any resource with item level change notification.
		+	* finally remove copy 1 and make copy 2 the new copy 1
		+
		+	=== Drawbacks ===
		+	* Two files are parsed on fileChanged()
		+
		+	=== Advantages ===
		+	* Reduced memory usage
		+	* if dirty items are not part of the change, no need to backup or panic ???
		+	* Better data consistency in case files are changed by non-Akonadi programs.