About Features Downloads Getting Started Documentation Events Support GitHub

Site Tools


indexing:tracking_record_changes

Tracking Record Changes

IMPORTANT: This page refers to features that were added in VuFind 1.1. If you are using an earlier version, you will have to upgrade.

Introduction

For certain types of functionality, VuFind needs to be aware not only of the content of the records in its index, but also the times at which these records were first indexed and most recently changed. By understanding the history of its own records, VuFind can provide RSS feeds that show recent changes, and it can export its contents using the popular OAI-PMH protocol.

Keeping track of the information needed to support these features adds some overhead – more data in your MySQL database, and slower indexing times. Because not everyone needs the features, they are disabled by default. This page explains how to activate them when you need them.

Enabling Change Tracking

The most important thing that you need to do in order to enable VuFind's change tracking is to edit the local MARC mappings file and uncomment the lines that fill in the “first_indexed” and “last_indexed” fields of your Solr index. If you are using VuFind 2.x or higher, this usually consists of:

1. Copy import/marc_local.properties into the import subdirectory of your local_settings_directory (usually $VUFIND_HOME/local/import), if you don't already have a local copy of marc_local.properties.

2. Uncomment the *_indexed lines in the local copy of marc_local.properties.

If you are indexing non-MARC records, some change tracking facilities also exist in the XSLT index tool. Depending on your data source, you may need to customize your XSLT and/or properties file to implement support.

After making the necessary configuration changes, you will need to reindex all of your records. This will take a long time, as it will be generating a database so that VuFind can correctly populate these fields in the future. “first_indexed” contains the time that the record was first indexed by VuFind, while “last_indexed” contains the time that the record was most recently changed. VuFind stores information in its database so that even if you upgrade Solr and destroy your entire index, it will still be able to rebuild the correct “first_indexed” and “last_indexed” values. In the future, when you reindex existing records, the process should be faster than the first time around, since for records that have not actually changed, VuFind will not have to do much extra database work.

A Note on Memory Usage

SolrMarc may require more memory in order to update the database and the Solr index at the same time. If you get error messages about running out of heap space during the index process, try editing import-marc.sh or import-marc.bat (depending on your platform) to change the Java memory settings. See the performance page for more details on Java memory tuning.

OAI-PMH Server Functionality

Location

VuFind's OAI-PMH server can be found at http://[your_server]/[your_vufind_base_path]/OAI/Server.

Requirements

The server will not return any records unless the first_indexed and last_indexed fields of the Solr index are properly filled in (see Enabling Change Tracking, above).

Configuration

Several aspects of the server (including ID prefixes, administrative email addresses and set behavior) can be configured through the [OAI] section of config.ini. See the comments in that file for details on how the settings work.

Metadata Formats

Out of the box, VuFind's OAI-PMH server supports basic oai_dc in a minimalistic way and is capable of exporting MARC records in XML format.

To add additional metadata formats, simply make changes to the getXML() method of the record driver(s) you are using.

Important: The server is currently coded to assume that all supported metadata formats are supported by all records in the index. In order to display only a subset of records based on the metadata format requested, it would be necessary to make some minor adjustments to the code and the underlying Solr index. If there is any demand for this feature, please open a JIRA ticket.

Why use OAI-PMH in VuFind?

Not everyone is going to want to expose their VuFind records through OAI-PMH, but in some cases it may be convenient – if VuFind is how you present your holdings to the world, it makes sense to offer the data to automated agents as well as to humans. If you have already gone through the complicated work of getting data out of your ILS and into VuFind, using OAI-PMH may be a way of saving you from having to repeat that effort to send records to other places.

If you have found this feature useful, feel free to add comments and advice here!

More on OAI-PMH

For a general introduction to the OAI-PMH protocol, see this tutorial.

Enhanced RSS Functionality

When record change tracking is enabled, RSS feeds will be more effective. Without tracking, RSS dates are based on publication dates, which may or may not correspond with the dates that items were added to your collection. With change tracking turned on, RSS feeds will be sorted by index date and will display index dates within the feed. This means that patrons can truly track searches in order to be notified of new additions within their chosen areas of interest.

Note that you can customize some of the RSS behavior in VuFind – see the [RSS] section of searches.ini.

Solr-driven New Items

:!: This feature was added in VuFind 2.3.

By default, VuFind tries to use the ILS driver to power the “new items” search. However, not all ILS drivers support this functionality. If you turn on record change tracking, you can modify the method setting to “solr” in the [NewItem] section of searches.ini to gain new item searches without an ILS dependency.

indexing/tracking_record_changes.txt · Last modified: 2016/08/30 12:26 by demiankatz