About Features Downloads Getting Started Documentation Events Support GitHub

Love VuFind®? Consider becoming a financial supporter. Your support helps build a better VuFind®!

Site Tools


Warning: This page has not been updated in over over a year and may be outdated or deprecated.
indexing:tracking_record_changes

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
tracking_record_changes [2015/12/14 16:58] demiankatzindexing:tracking_record_changes [2021/08/03 13:59] (current) – [More on OAI-PMH] demiankatz
Line 13: Line 13:
 ===== Enabling Change Tracking ====== ===== Enabling Change Tracking ======
  
-The most important thing that you need to do in order to enable VuFind's change tracking is to edit the [[indexing:solrmarc:local_marc_mappings|local MARC mappings]] file and uncomment the lines that fill in the "first_indexed" and "last_indexed" fields of your Solr index.+The most important thing that you need to do in order to enable VuFind's change tracking is to edit the [[indexing:solrmarc:local_marc_mappings|local MARC mappings]] file and uncomment the lines that fill in the "first_indexed" and "last_indexed" fields of your Solr index. If you are using VuFind 2.x or higher, this usually consists of:
  
-If you are indexing non-MARC records, some change tracking facilities also exist in the [[indexing:xml|XSLT index tool]].  For examplethe VuDL importer supports change tracking if you edit [[https://vufind.svn.sourceforge.net/svnroot/vufind/trunk/import/vudl.properties|import/vudl.properties]].+1. Copy import/marc_local.properties into the import subdirectory of your [[configuration:local_settings_directory]] (usually $VUFIND_HOME/local/import), if you don't already have a local copy of marc_local.properties.
  
-After making the necessary configuration changes, you will need to reindex all of your records.  This will take a long time, as it will be generating a database so that VuFind can correctly populate these fields in the future.  "first_indexed" contains the time that the record was first indexed by VuFind, while "last_indexed" contains the time that the record was most recently changed.  VuFind stores information in its database so that even if you upgrade Solr and destroy your entire index, it will still be able to rebuild the correct "first_indexed" and "last_indexed" values.  In the future, when you reindex existing records, the process should be faster than the first time around, since for records that have not actually changed, VuFind will not have to do much extra database work.+2. Uncomment the *_indexed lines in the local copy of marc_local.properties. The example lines assume that your record IDs come from the 001 field; if you use IDs in a different place, change 001 to an appropriate pattern (e.g. 999c for Koha). 
 + 
 +If you are indexing non-MARC records, some change tracking facilities also exist in the [[indexing:xml|XSLT index tool]]. Depending on your data source, you may need to customize your XSLT and/or properties file to implement support. 
 + 
 +After making the necessary configuration changes, you will need to reindex all of your records.  This will take a long time, as it will be generating a database so that VuFind can correctly populate these fields in the future.  "first_indexed" contains the time that the record was first indexed by VuFind, while "last_indexed" contains the time that the record was most recently changed. For consistency and appropriate OAI-PMH behavior, it is STRONGLY RECOMMENDED that all times be stored in UTC; VuFind's provided tools will already do this, but if you build your own custom indexer, you should conform to this standard. VuFind stores information in its database so that even if you upgrade Solr and destroy your entire index, it will still be able to rebuild the correct "first_indexed" and "last_indexed" values.  In the future, when you reindex existing records, the process should be faster than the first time around, since for records that have not actually changed, VuFind will not have to do much extra database work.
  
 ==== A Note on Memory Usage ==== ==== A Note on Memory Usage ====
  
-[[indexing:solrmarc|SolrMarc]] may require more memory in order to update the database and the Solr index at the same time.  If you get error messages about running out of heap space during the index process, try editing import-marc.sh or import-marc.bat (depending on your platform) to change the Java memory settings.  See the [[Performance]] page for more details on Java memory tuning.+[[indexing:solrmarc|SolrMarc]] may require more memory in order to update the database and the Solr index at the same time.  If you get error messages about running out of heap space during the index process, try editing import-marc.sh or import-marc.bat (depending on your platform) to change the Java memory settings.  See the [[administration:performance]] page for more details on Java memory tuning.
  
  
Line 37: Line 41:
 ==== Configuration ==== ==== Configuration ====
  
-Several aspects of the server (including ID prefixes, administrative email addresses and set behavior) can be configured through the [OAI] section of [[config.ini]].  See the comments in that file for details on how the settings work.+Several aspects of the server (including ID prefixes, administrative email addresses and set behavior) can be configured through the [OAI] section of [[configuration:files:config.ini]].  See the comments in that file for details on how the settings work.
  
 ==== Metadata Formats ==== ==== Metadata Formats ====
Line 57: Line 61:
 For a general introduction to the OAI-PMH protocol, see [[http://www.oaforum.org/tutorial/|this tutorial]]. For a general introduction to the OAI-PMH protocol, see [[http://www.oaforum.org/tutorial/|this tutorial]].
  
 +==== Related Video ====
  
 +For more on OAI-PMH, see the [[videos:oai-pmh_server_and_harvest_functionality|OAI-PMH Server and Harvest Functionality]] video.
 ===== Enhanced RSS Functionality ===== ===== Enhanced RSS Functionality =====
  
 When record change tracking is enabled, RSS feeds will be more effective.  Without tracking, RSS dates are based on publication dates, which may or may not correspond with the dates that items were added to your collection.  With change tracking turned on, RSS feeds will be sorted by index date and will display index dates within the feed.  This means that patrons can truly track searches in order to be notified of new additions within their chosen areas of interest. When record change tracking is enabled, RSS feeds will be more effective.  Without tracking, RSS dates are based on publication dates, which may or may not correspond with the dates that items were added to your collection.  With change tracking turned on, RSS feeds will be sorted by index date and will display index dates within the feed.  This means that patrons can truly track searches in order to be notified of new additions within their chosen areas of interest.
  
-Note that you can customize some of the RSS behavior in VuFind -- see the [RSS] section of [[searches.ini]].+Note that you can customize some of the RSS behavior in VuFind -- see the [RSS] section of [[configuration:files:searches.ini]]
 + 
 +===== Solr-driven New Items ===== 
 + 
 +:!: This feature was added in VuFind 2.3. 
 + 
 +By default, VuFind tries to use the ILS driver to power the "new items" search. However, not all ILS drivers support this functionality. If you turn on record change tracking, you can modify the method setting to "solr" in the [NewItem] section of [[configuration:files:searches.ini|searches.ini]] to gain new item searches without an ILS dependency. 
 + 
 +===== Frequently Asked Questions ===== 
 + 
 +This section contains answers to frequently asked questions about the technical details of the change-tracking implementation. 
 + 
 +==== If I reindex my existing records, will VuFind think that they are new? ==== 
 + 
 +No, the change tracking system will only update dates in its internal database if it encounters a record ID it has never seen before, or if the record itself has changed. If you index the same MARC file over and over again, that record will only be recorded in the database once. However, if you edit the MARC record and export a new version, when you index the changed record, VuFind will update the "last indexed" value to reflect that something is different. 
 + 
 +This behavior is dependent on records containing last-changed dates. In the case of MARC records, VuFind checks the 005 and 008 fields for modification dates. Note that if no dates are found, VuFind will always assume that the record has changed as a precaution; thus, it is important to ensure that your ILS is including one of these fields in exported records if you want reliable change tracking. In the case of XSLT records, the custom PHP methods used by the XSLT expect a modification date parameter, and you should be sure to extract it from an appropriate place in your XML to ensure consistent behavior. 
 + 
 +==== What happens when records are deleted? ==== 
 + 
 +If you remove records from your ILS, you should inform VuFind about the deleted records so it can update its change tracking database. This enables the system to correctly report deleted records through OAI-PMH, and it also allows the system to treat records as new if they later are added again after being previously deleted. You can use the util/deletes [[administration:command_line_utilities|command line utility]] for this purpose. 
 + 
 +For systems that have no easy way of reporting deleted records, a common solution is to simply delete VuFind's index, and then reindex from a fresh full export. If you do this, the deleted records will be removed from VuFind's index correctly, so normal search behavior will work as expected. However, the change tracking system will not be aware that records have been removed, and the now-missing records will not be reported as deleted through OAI-PMH (or any other feature that might rely on delete tracking functionality). This may or may not be a problem for you, depending on your use case.
 ---- struct data ---- ---- struct data ----
 +properties.Page Owner : 
 ---- ----
  
indexing/tracking_record_changes.1450112318.txt.gz · Last modified: 2015/12/14 16:58 by demiankatz