About Features Downloads Getting Started Documentation Events Support GitHub

Love VuFind®? Consider becoming a financial supporter. Your support helps build a better VuFind®!

Site Tools


Warning: This page has not been updated in over over a year and may be outdated or deprecated.
indexing:deduplication

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
deduplication [2014/01/02 20:40] – [Sample datasources.ini] demiankatzindexing:deduplication [2023/03/20 16:33] (current) – [Search Process] demiankatz
Line 1: Line 1:
 ====== Support for Deduplication ====== ====== Support for Deduplication ======
  
-// The features described on this page are available starting with VuFind 2.3. //+// The features described on this page are available starting with VuFind® 2.3. // 
 + 
 +===== Introduction ===== 
 + 
 +The deduplication feature allows multiple records from multiple sources to be combined together and displayed as a single search result. It requires some extra setup and external tools to fully implement. Users who choose not to use deduplication may instead wish to configure the [[configuration:record_versions|Record Versions]] feature, which offers a different method for associating related records. (It is also possible for the Deduplication and Record Versions features to be used to complement one another).
  
 ===== Solr Setup ===== ===== Solr Setup =====
  
-VuFind includes support for displaying deduplicated records. This requires that records are deduplicated before indexing into Solr, and that a so called merged record is created for each dedup group (group of original duplicate records) alongside the original records. [[https://github.com/KDK-Alli/RecordManager|RecordManager]] can be used for deduplication as it has built-in support for VuFind-compatible deduplication, but VuFind doesn't require RecordManager to be used, just some index fields and the merged record to be present.+VuFind® includes support for displaying deduplicated records. This requires that records are deduplicated before indexing into Solr, and that a so called merged record is created for each dedup group (group of original duplicate records) alongside the original records. 
 + 
 +==== RecordManager ==== 
 + 
 +[[https://github.com/KDK-Alli/RecordManager|RecordManager]] can be used for deduplication as it has built-in support for VuFind®-compatible deduplication, but VuFind® doesn't require RecordManager to be used, just some index fields and the merged record to be present. 
 + 
 +Using RecordManager does offer some specific advantages:
  
 +  - RecordManager can find the best record among the deduplicated records to use as the base record when creating a merged record.
 +  - There's control over how single-valued and multi-valued fields are merged into the merged record, as well as the possibility of handling first_indexed and last_indexed data.
 +  - The records belonging to a dedup group can also be enriched with data from the merged record, so enrichment can be achieved in both directions.
 +  - The mechanism can ensure e.g. that two records from the same data source never get deduplicated. This is a built-in assumption in RecordManager's default deduplication algorithm that records in a single source should already be distinct ones.
 ==== Required Solr Fields in Merged Records ==== ==== Required Solr Fields in Merged Records ====
  
Line 12: Line 26:
   * **merged_boolean** - A boolean field with value "true" to indicate that this is a merged record.   * **merged_boolean** - A boolean field with value "true" to indicate that this is a merged record.
      
-==== Required Solr Fields in Original Records ====+==== Required Solr Fields in Original Records with Duplicates ====
  
-  * **merged_child_boolean** - A boolean field with value "true" to indicate that this is an original record belonging to a dedup group.+  * **merged_child_boolean** - A boolean field with value "true" to indicate that this is an original record belonging to a dedup group. Leave this out from records that don't have any duplicates.
  
 ==== Sample Records ==== ==== Sample Records ====
Line 60: Line 74:
 |                  | testsrc.21346 | |                  | testsrc.21346 |
  
 +=== Record With No Duplicates ===
 +
 +^ field    ^ contents        ^
 +| id       | testsrc.123     |
 +| title    | Network science illustrated |
 +| language | eng             |
 +| publishDate | 2012         |
 +| topic    | Computer networks |
 +|          | Electronics in military engineering |
 +|          | Military engineering |
  
 ===== Search Process ===== ===== Search Process =====
Line 65: Line 89:
 Records with merged_child_boolean=true are filtered from the results during the initial Solr search. Then the preferred original record is selected from each merge record found, and the merge record replaced with the original record. Information on all the records belonging to the dedup group is added to the original records in "dedup_data" field so that this information can be displayed to the user e.g. with links to other records. The preferred record is always first in "dedup_data". Records with merged_child_boolean=true are filtered from the results during the initial Solr search. Then the preferred original record is selected from each merge record found, and the merge record replaced with the original record. Information on all the records belonging to the dedup group is added to the original records in "dedup_data" field so that this information can be displayed to the user e.g. with links to other records. The preferred record is always first in "dedup_data".
  
 +==== Architecture Note: Field Collapsing / Collapse/Expand ====
 +
 +Note that while Solr supports features such as "field collapsing" and "collapse/expand" which could be used to achieve similar deduplication behavior, the deduplication mechanism here does not utilize these features. This avoids the performance cost associated with such functionality, and also allows broader search results. Collapse/expand only works for a search result set. VuFind®'s deduplication doesn't require all the records in the group to match the search terms. It's enough that the merge record does. This may or may not be important depending on how things are done, but at least it allows one to present the "best" result record in search results without having to re-merge anything.
 ===== Configuration ===== ===== Configuration =====
  
-The following settings in the Records section of [[searches.ini]] affect deduplication:+The following settings in the Records section of [[configuration:files:searches.ini]] affect deduplication:
  
   * **deduplication** Whether support for deduplicated records is enabled   * **deduplication** Whether support for deduplicated records is enabled
   * **record_sources** Optional: A comma-separated list of record sources (ID prefixes separated from the actual record id with a period) in order of precedence. The sooner the source is found in the list, the higher the priority when selecting a preferred record to display.   * **record_sources** Optional: A comma-separated list of record sources (ID prefixes separated from the actual record id with a period) in order of precedence. The sooner the source is found in the list, the higher the priority when selecting a preferred record to display.
  
-There is also an optional datasources.ini that can be used to map institutions to source prefixes so that the correct record can be selected when the building facet is used. Each section in datasources.ini, which is compatible with [[https://github.com/KDK-Alli/RecordManager/wiki/Configuration#data-source-settings|RecordManager's datasources.ini]] resembles source id. The institution setting in each section is matched to the selected building facet, and the section name is used to find the record priority from **record_sources**.+There is also an optional datasources.ini that can be used to map institutions to source prefixes so that the correct record can be selected when the building facet is used. This mapping is needed unless the building facet values are identical to the source id's. Each section in datasources.ini, which is compatible with [[https://github.com/KDK-Alli/RecordManager/wiki/Configuration#data-source-settings|RecordManager's datasources.ini]]resembles source id. The institution setting in each section is matched to the selected building facet, and the section name is used to find the record priority from **record_sources**. 
  
 Source strings can be translated with source_ prefix. An example translation in en.ini could be: Source strings can be translated with source_ prefix. An example translation in en.ini could be:
Line 88: Line 115:
 </code> </code>
 ---- struct data ---- ---- struct data ----
 +properties.Page Owner : emaijala
 ---- ----
  
indexing/deduplication.1388695253.txt.gz · Last modified: 2014/06/13 13:12 (external edit)