Development/Tutorials/Writing file analyzers - Revision history

Neverendingo: Text replace - "" to ""

2011-06-30T10:20:13Z

Text replace - "<syntaxhighlight lang="make">" to "<syntaxhighlight lang="cmake">"

Neverendingo: /* Building the analyzer */

2011-06-29T19:58:25Z

‎Building the analyzer

Neverendingo: Text replace - "" to ""

2011-06-29T19:42:24Z

Text replace - "</code>" to "</syntaxhighlight>"

Neverendingo: Text replace - "`" to ""`


2011-06-29T19:38:54Z
Text replace - "<code cpp>" to "<syntaxhighlight lang="cpp">"

				
				
				
				
			
			← Older revision
			Revision as of 19:38, 29 June 2011
			
Line 48:
Line 48:
  Here is what a class used to process BMP images might look like:   Here is what a class used to process BMP images might look like:
    
− <code cpp> + <syntaxhighlight lang="cpp">
  #include <strigi/streamendanalyzer.h>   #include <strigi/streamendanalyzer.h>
    
Line 74:
Line 74:
  It is by no means necessary for <tt>checkHeader</tt> to be 100% correct. The most important thing is that it should allow one to quickly determine if an analyzer  can not handle a resource. Should it by accident return true and thus indicate that a resource can be handled, then it can always handle this in the <tt>analyze()</tt> method.   It is by no means necessary for <tt>checkHeader</tt> to be 100% correct. The most important thing is that it should allow one to quickly determine if an analyzer  can not handle a resource. Should it by accident return true and thus indicate that a resource can be handled, then it can always handle this in the <tt>analyze()</tt> method.
    
− <code cpp> + <syntaxhighlight lang="cpp">
  bool BmpEndAnalyzer::checkHeader( const char* header,   bool BmpEndAnalyzer::checkHeader( const char* header,
                                     int32_t headersize) const                                      int32_t headersize) const
Line 94:
Line 94:
    
  If a resource passes this test, the <tt>analyze()</tt> function will be called. The real work occurs in this function. In this tutorial, we will not do a complete analysis, but we only look at the way the pixels are stored in the BMP file. This information is stored in the bytes 30-33 that encode a number.   If a resource passes this test, the <tt>analyze()</tt> function will be called. The real work occurs in this function. In this tutorial, we will not do a complete analysis, but we only look at the way the pixels are stored in the BMP file. This information is stored in the bytes 30-33 that encode a number.
− <code cpp> + <syntaxhighlight lang="cpp">
  using namespace Strigi;   using namespace Strigi;
  char BmpEndAnalyzer::analyze(AnalysisResult& idx, InputStream* in)   char BmpEndAnalyzer::analyze(AnalysisResult& idx, InputStream* in)
Line 137:
Line 137:
    
  We will now look at how to implement a StreamThroughAnalyzer. For this, we need to implement three functions.   We will now look at how to implement a StreamThroughAnalyzer. For this, we need to implement three functions.
− <code cpp> + <syntaxhighlight lang="cpp">
  class BmpThroughAnalyzer : public Strigi::StreamThroughAnalyzer {   class BmpThroughAnalyzer : public Strigi::StreamThroughAnalyzer {
  private:   private:
Line 153:
Line 153:
  For simple file formats where all information is in the initial part of the file, all of the work gets done in connectInputStream(). For more complicated cases look at other examples, such as DigestThroughAnalyzer.   For simple file formats where all information is in the initial part of the file, all of the work gets done in connectInputStream(). For more complicated cases look at other examples, such as DigestThroughAnalyzer.
  In connectInputStream(), we perform the same analysis as we did in the StreamEndAnalyzer:   In connectInputStream(), we perform the same analysis as we did in the StreamEndAnalyzer:
− <code cpp>Strigi::InputStream + <syntaxhighlight lang="cpp">Strigi::InputStream
  BmpThroughAnalyzer::connectInputStream(Strigi::InputStream* in) {    // read compression type (bytes #30-33)   BmpThroughAnalyzer::connectInputStream(Strigi::InputStream* in) {    // read compression type (bytes #30-33)
       const char* h;        const char* h;
Line 195:
Line 195:
    
  The AnalyzerFactoryFactory is used only in the loadable plugins. It is not needed for analyzers that are part of the Strigi core. To initialize the StreamThroughAnalyzerFactories and StreamEndAnalyzerFactories in plugins, we need to implement an AnalyzerFactoryFactory. We do so by implementing one or two functions: streamThroughAnalyzerFactories() and streamEndAnalyzerFactories(). This function returns instances of all the factories available in a plugin. Here is for example the AnalyzerFactoryFactory for the KDE trash file analyzer (we cannot use the BMP analyzer here, because it is in Strigi core):   The AnalyzerFactoryFactory is used only in the loadable plugins. It is not needed for analyzers that are part of the Strigi core. To initialize the StreamThroughAnalyzerFactories and StreamEndAnalyzerFactories in plugins, we need to implement an AnalyzerFactoryFactory. We do so by implementing one or two functions: streamThroughAnalyzerFactories() and streamEndAnalyzerFactories(). This function returns instances of all the factories available in a plugin. Here is for example the AnalyzerFactoryFactory for the KDE trash file analyzer (we cannot use the BMP analyzer here, because it is in Strigi core):
− <code cpp> + <syntaxhighlight lang="cpp">
  class Factory : public AnalyzerFactoryFactory {   class Factory : public AnalyzerFactoryFactory {
  public:   public:
Line 216:
Line 216:
  The class BmpEndAnalyzerFactory looks like this:   The class BmpEndAnalyzerFactory looks like this:
    
− <code cpp> + <syntaxhighlight lang="cpp">
  class BmpEndAnalyzerFactory : public Strigi::StreamEndAnalyzerFactory {   class BmpEndAnalyzerFactory : public Strigi::StreamEndAnalyzerFactory {
  friend class BmpEndAnalyzer;   friend class BmpEndAnalyzer;
Line 238:
Line 238:
  But first we need to register the fields:   But first we need to register the fields:
    
− <code cpp> + <syntaxhighlight lang="cpp">
  #include <strigi/fieldtypes.h>   #include <strigi/fieldtypes.h>
  #include <strigi/analysisresult.h>   #include <strigi/analysisresult.h>
Line 255:
Line 255:
    
  The pointers to the registered fields are used during the analysis to identify the type of data we have analyzed:   The pointers to the registered fields are used during the analysis to identify the type of data we have analyzed:
− <code cpp> + <syntaxhighlight lang="cpp">
      idx.addValue(factory->typeField, "OS/2 Color Icon");       idx.addValue(factory->typeField, "OS/2 Color Icon");
  </code>   </code>



Captainbeah: fix cmake variable name
2008-01-05T23:16:07Z
fix cmake variable name

				
				
				
				
			
			← Older revision
			Revision as of 23:16, 5 January 2008
			
Line 268:
Line 268:
  <code>   <code>
  add_library(trash MODULE trashthroughanalyzer.cpp trashimpl.cpp)   add_library(trash MODULE trashthroughanalyzer.cpp trashimpl.cpp)
− target_link_libraries(trash ${STREAMANALYZER_LIBRARY}) + target_link_libraries(trash ${STRIGI_STREAMANALYZER_LIBRARY})
  set_target_properties(trash PROPERTIES   set_target_properties(trash PROPERTIES
       PREFIX strigita_)        PREFIX strigita_)



Tampakrap: Added other analyzer types, just to remind they exist :)
2007-06-25T17:39:52Z
Added other analyzer types, just to remind they exist :)

				
				
				
				
			
			← Older revision
			Revision as of 17:39, 25 June 2007
			
Line 33:
Line 33:
  == Choosing the type of analyzer ==   == Choosing the type of analyzer ==
    
− There are two types of analyzers: StreamThroughAnalyzer and StreamEndAnalyzer. The latter is more powerful and a bit easier to program, but has a limitation: only one StreamEndAnalyzer can analyze a particular resource, while you can use as many StreamThroughAnalyzers as you like. Most analyzers can be written as StreamThroughAnalyzers. The most important exception is for analyzers that extract embedded resources from a stream. Examples of this are the ZipEndAnalyzer, the MailEndAnalyzer and the RpmEndAnalyzer. + There are two main types of analyzers: '''StreamThroughAnalyzer''' and '''StreamEndAnalyzer'''. The latter is more powerful and a bit easier to program, but has a limitation: only one StreamEndAnalyzer can analyze a particular resource, while you can use as many StreamThroughAnalyzers as you like. Most analyzers can be written as StreamThroughAnalyzers. The most important exception is for analyzers that extract embedded resources from a stream. Examples of this are the ZipEndAnalyzer, the MailEndAnalyzer and the RpmEndAnalyzer.
    
  In this tutorial we focus on a simple example file type: BMP images. The information we will get from this file is located at the start of the file. It turns out that in this case, it is just as easy to implement the analyzer as a StreamEndAnalyzer as a StreamThroughAnalyzer. We will implement it as a StreamEndAnalyzer and point out how to do the same as a StreamThroughAnalyzer.   In this tutorial we focus on a simple example file type: BMP images. The information we will get from this file is located at the start of the file. It turns out that in this case, it is just as easy to implement the analyzer as a StreamEndAnalyzer as a StreamThroughAnalyzer. We will implement it as a StreamEndAnalyzer and point out how to do the same as a StreamThroughAnalyzer.
  + 
  + Two other types of stream analyzers have been added to Strigi. '''StreamLineAnalyzer''', for file format based on lines of plan text, '''StreamSaxAnalyzer''', for XML based file such as SVG.
    
  == StreamEndAnalyzer ==   == StreamEndAnalyzer ==



Tampakrap: add link to existing hachoir parser code (from pvandewyngaerde)
2007-04-21T22:41:59Z
add link to existing hachoir parser code (from pvandewyngaerde)

				
				
				
				
			
			← Older revision
			Revision as of 22:41, 21 April 2007
			
Line 26:
Line 26:
    
  == Look for existing code ==   == Look for existing code ==
− If you want to see some code examples, take a look at the already implemented file analyzers at [http://websvn.kde.org/trunk/kdesupport/strigi/src/streamanalyzer/ /kdesupport/strigi/src/streamanalyzer/] + If you want to see some code examples, take a look at the already implemented file analyzers at [http://websvn.kde.org/trunk/kdesupport/strigi/src/streamanalyzer/ /kdesupport/strigi/src/streamanalyzer/].
  +  
  + Some examples of meta-data extraction from files can also be found in the 
  + [http://www.hachoir.org Hachoir project] in the [http://hachoir.org/browser/trunk/hachoir-parser/hachoir_parser online parser sourcecode].
    
  == Choosing the type of analyzer ==   == Choosing the type of analyzer ==





Tampakrap: /* Choosing the type of analyzer */ Fixed an error
2007-04-13T19:39:43Z
‎Choosing the type of analyzer:  Fixed an error

				
				
				
				
			
			← Older revision
			Revision as of 19:39, 13 April 2007
			
Line 30:
Line 30:
  == Choosing the type of analyzer ==   == Choosing the type of analyzer ==
    
− There are two types of analyzers: StreamThroughAnalyzer and StreamEndAnalyzer. The latter is more powerful and a bit easier to program, but has a limition: only one StreamEndAnalyzer can analyze a particular resource, while you can use as many StreamThroughAnalyzers as you like. Most analyzers can be written as StreamThroughAnalyzers. The most important exception is for analyzers that extract embedded resources from a stream. Examples of this are the ZipEndAnalyzer, the MailEndAnalyzer and the RpmEndAnalyzer. + There are two types of analyzers: StreamThroughAnalyzer and StreamEndAnalyzer. The latter is more powerful and a bit easier to program, but has a limitation: only one StreamEndAnalyzer can analyze a particular resource, while you can use as many StreamThroughAnalyzers as you like. Most analyzers can be written as StreamThroughAnalyzers. The most important exception is for analyzers that extract embedded resources from a stream. Examples of this are the ZipEndAnalyzer, the MailEndAnalyzer and the RpmEndAnalyzer.
    
  In this tutorial we focus on a simple example file type: BMP images. The information we will get from this file is located at the start of the file. It turns out that in this case, it is just as easy to implement the analyzer as a StreamEndAnalyzer as a StreamThroughAnalyzer. We will implement it as a StreamEndAnalyzer and point out how to do the same as a StreamThroughAnalyzer.   In this tutorial we focus on a simple example file type: BMP images. The information we will get from this file is located at the start of the file. It turns out that in this case, it is just as easy to implement the analyzer as a StreamEndAnalyzer as a StreamThroughAnalyzer. We will implement it as a StreamEndAnalyzer and point out how to do the same as a StreamThroughAnalyzer.



Pippin: /* AnalyzerFactoryFactory */ missed a bit before
2007-03-31T17:04:13Z
‎AnalyzerFactoryFactory:  missed a bit before

				
				
				
				
			
			← Older revision
			Revision as of 17:04, 31 March 2007
			
Line 189:
Line 189:
  === AnalyzerFactoryFactory ===   === AnalyzerFactoryFactory ===
    
− The AnalyzerFactoryFactory is used only in the loadable plugins. It is not needed for analyzers that are part of the Strigi core. To initialize the StreamThroughAnalyzerFactories and StreamEndAnalyzerFactories in plugins, we need to implement an AnalyzerFactoryFactory. We do so by implementing one or two functions: getStreamThroughAnalyzerFactories() and getStreamEndAnalyzerFactories(). This function returns instances of all the factories available in a plugin. Here is for example the AnalyzerFactoryFactory for the KDE trash file analyzer (we cannot use the BMP analyzer here, because it is in Strigi core): + The AnalyzerFactoryFactory is used only in the loadable plugins. It is not needed for analyzers that are part of the Strigi core. To initialize the StreamThroughAnalyzerFactories and StreamEndAnalyzerFactories in plugins, we need to implement an AnalyzerFactoryFactory. We do so by implementing one or two functions: streamThroughAnalyzerFactories() and streamEndAnalyzerFactories(). This function returns instances of all the factories available in a plugin. Here is for example the AnalyzerFactoryFactory for the KDE trash file analyzer (we cannot use the BMP analyzer here, because it is in Strigi core):
  <code cpp>   <code cpp>
  class Factory : public AnalyzerFactoryFactory {   class Factory : public AnalyzerFactoryFactory {



Pippin: Update to reflect new Strigi API
2007-03-31T17:03:04Z
Update to reflect new Strigi API
Show changes

@@ Line 64: / Line 64: @@
      const BmpEndAnalyzerFactory* factory;
 };
-</code>
+</syntaxhighlight>
 The <tt>factory</tt> object is used to load the analyzer at runtime and will be covered in detail later in this tutorial.
@@ Line 89: / Line 89: @@
      return ok;
+}
-</code>
+</syntaxhighlight>
 A BMP file can start with six different initial bytes. If the header matches any of them, we return true.
@@ Line 129: / Line 129: @@
      return Ok;
+}
-</code>
+</syntaxhighlight>
 First we read exactly 34 bytes from the stream. We do not need to allocate a buffer; this is handled by the stream. The stream returns a pointer to its internal buffer, which avoids data copying and buffer allocation. If the resource has less than 34 bytes or if an error occurred during reading, we return with an error code.
@@ Line 149: / Line 149: @@
      bool isReadyWithStream() { return true; }
 };
-</code>
+</syntaxhighlight>
 For simple file formats where all information is in the initial part of the file, all of the work gets done in connectInputStream(). For more complicated cases look at other examples, such as DigestThroughAnalyzer.
@@ Line 181: / Line 181: @@
      return in;
+}
-</code>
+</syntaxhighlight>
 The difference with StreamEndAnalyzer is that we must make sure we only return a stream that has the current position at the start of the stream. We can ensure this by calling reset(0) after each call to read(). This will not affect the pointer to the character data that was set in the call to read().
@@ Line 208: / Line 208: @@
 // macro that initializes the Factory when the plugin is loaded
 STRIGI_ANALYZER_FACTORY(Factory)
-</code>
+</syntaxhighlight>
 === StreamEndAnalyzerFactory ===
@@ Line 232: / Line 232: @@
      void registerFields(Strigi::FieldRegister&);
 };
-</code>
+</syntaxhighlight>
 All members are private, which is ok, because the important functions are virtual and thus accessible anyway. The functions name() and newInstance() are selfexplanatory. The other important function is registerFields(Strigi::FieldRegister&). To speed up the extraction of fields, we dont use strings to identify fields, but we use pointers to registered fields. These RegisteredField instances are stored in a global register. When you extract a piece of metadata, you pass the pointer to the registered field to identify the metadata.
@@ Line 250: / Line 250: @@
 , 0);
+}
-</code>
+</syntaxhighlight>
 We pass the key of the field to the register, along with it's type, the maximum number of times the field occurs per resource and the parent of the field. The parent of a song tile, for example, could be a more general title field. The datatype of this field should be the same or a subset of the fieldtype of the parent.
@@ Line 257: / Line 257: @@
 <syntaxhighlight lang="cpp">
     idx.addValue(factory->typeField, "OS/2 Color Icon");
-</code>
+</syntaxhighlight>
 == Building the analyzer ==
@@ Line 272: / Line 272: @@
      PREFIX strigita_)
 install(TARGETS trash LIBRARY DESTINATION ${LIB_INSTALL_DIR}/strigi)
-</code>
+</syntaxhighlight>
 == Testing your code ==

@@ Line 266: / Line 266: @@
 * Install the plugin in the lib/strigi directory.
 Here is the CMakeLists.txt code to do this:
-<syntaxhighlight lang="make">
+<syntaxhighlight lang="cmake">
 add_library(trash MODULE trashthroughanalyzer.cpp trashimpl.cpp)
 target_link_libraries(trash ${STRIGI_STREAMANALYZER_LIBRARY})