About Features Downloads Getting Started Documentation Events Support GitHub

Love VuFind®? Consider becoming a financial supporter. Your support helps build a better VuFind®!

Site Tools


Warning: This page has not been updated in over over a year and may be outdated or deprecated.
development:howtos:connecting_a_new_external_data_source

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
vufind2:connecting_a_new_external_data_source [2012/09/11 14:10] demiankatzdevelopment:howtos:connecting_a_new_external_data_source [2023/06/19 18:21] (current) – [6. Create Controllers] demiankatz
Line 1: Line 1:
 ====== Connecting a New External Data Source ====== ====== Connecting a New External Data Source ======
  
-VuFind includes support for several external services in addition to the core Solr index.  However, it may sometimes be necessary to build support for a new service.  This page will help take you through the process.+VuFind® includes support for several external services in addition to the core Solr index.  However, it may sometimes be necessary to build support for a new service.  This page will help take you through the process.
  
 ===== Initial Decisions ===== ===== Initial Decisions =====
Line 8: Line 8:
  
   * What URLs do you want to use for searches and record views?  This will affect the names of the routes and controllers you create. (i.e. http://localhost/vufind/WorldCat/Search, http://localhost/vufind/WorldCatRecord/12345)   * What URLs do you want to use for searches and record views?  This will affect the names of the routes and controllers you create. (i.e. http://localhost/vufind/WorldCat/Search, http://localhost/vufind/WorldCatRecord/12345)
-  * What "source" value do you want VuFind to use for identifying records from your new source?  This value will be saved in the database and passed in some URLs.  It should be short, unique and alphanumeric (i.e. "WorldCat", "Summon").+  * What "source" value do you want VuFind® to use for identifying records from your new source?  This value will be saved in the database and passed in some URLs.  It should be short, unique and alphanumeric (i.e. "WorldCat", "Summon").
   * What name do you want to use for your family of search classes; this should almost always be the same as your "source" value.   * What name do you want to use for your family of search classes; this should almost always be the same as your "source" value.
   * What name do you want to use for your record driver?  Again, this should usually be consistent with the "source" value.   * What name do you want to use for your record driver?  Again, this should usually be consistent with the "source" value.
Line 14: Line 14:
 ===== Step-by-Step Building Instructions ===== ===== Step-by-Step Building Instructions =====
  
-If you haven't already, you may want to read the [[Data Model / Key Concepts]] page for background information on how VuFind's pieces fit together.+If you haven't already, you may want to read the [[development:architecture:data_model_key_concepts|Data Model / Key Concepts]] page for background information on how VuFind®'s pieces fit together.
  
-Once you are ready to write code, you can follow these steps to get a new module added to VuFind.  Obviously, this is just a recommendation -- some of these steps can be taken out of order without any problems.+Once you are ready to write code, you can follow these steps to get a new module added to VuFind®.  Obviously, this is just a recommendation -- some of these steps can be taken out of order without any problems.
  
 Note that for examples below, we will assume that you have chosen Source, search class and record driver names of "Sample."  Just replace "Sample" with your actual choice. Note that for examples below, we will assume that you have chosen Source, search class and record driver names of "Sample."  Just replace "Sample" with your actual choice.
  
-The examples also assume that you are adding to core VuFind code.  You also have the option of building your classes inside a [[customizing_vufind_2.0#modules|custom module]]; to do this, you just need to create all new classes within your module's namespace and register them in your module's config/module.config.php instead of inside the main VuFind module.+The examples also assume that you are adding to core VuFind® code.  You also have the option of building your classes inside a [[development:architecture:customizing_vufind#modules|custom module]]; to do this, you just need to create all new classes within your module's namespace and register them in your module's config/module.config.php instead of inside the main VuFind® module.
  
-  - **Build/Find Connection Library** - Before you can integrate a new service into VuFind, you need to be able to talk to it from within PHP.  It's possible that the service provides its own library, which may save you some time.  Otherwise, you'll have to build something.  If you can solve your service-specific connection problems prior to VuFind integration, then you won't have as many different problems to debug during integration. +====1. Build Search Connector/Backend==== 
-  - **Implement Record Driver** - The record driver object is responsible for extracting key pieces of data about a record from the data returned by your chosen service.  To create it, follow these steps: +Before you can integrate a new service into VuFind®, you need to be able to talk to it from within PHP.  It's possible that the service provides its own library, which may save you some time.  Whether or not there is a pre-existing library, you should implement a search backend for VuFind®'s [[development:architecture:search_service|search service]].  This consists of a few steps: 
-    - Create a [[record drivers|record driver plugin]] for your data source -- i.e. module/VuFind/src/VuFind/RecordDriver/Sample.php containing a class called \VuFind\RecordDriver\Sample. +    - Create a backend implementing \VuFindSearch\Backend\BackendInterface (and possibly also some of the feature interfaces found in the \VuFindSearch\Feature namespace, if appropriate).  The purpose of this is to provide low-level functionality for performing searches and retrieving records.  Several example backend implementations can be found in the VuFindSearch module -- you should be able to use these as models during development.  In some cases (as in connecting to a custom Solr core) you may be able to use existing code but simply configure it differently. 
-    - Decide which existing record driver class to extend.  If the records you are working with are very distinctive, you will probably want to extend \VuFind\RecordDriver\AbstractBase and build custom templates.  However, in many cases, records will contain bibliographic data that is similar enough to VuFind's default Solr records to allow you to extend \VuFind\RecordDriver\SolrDefault (or one of its subclasses) and thus recycle many of VuFind's default built-in templates.  This approach is recommended whenever it is possible.+    - Create a factory to construct your backend in the context of VuFind®; some example factories can be found in the \VuFind\Search\Factory namespace. 
 +    - Register your factory by modifying the [[https://github.com/vufind-org/vufind/blob/dev/module/VuFind/src/VuFind/Search/BackendRegistry.php|\VuFind\Search\BackendRegistry]] class (if you are building a new feature to add to the core code) or by adding configuration to the ['vufind']['plugin_managers']['search_backend']['factories'] section of module.config.php (if you are building this in a local custom module).  The name of the registered service should match the "source" value you chose above. 
 + 
 +====2. Implement Record Driver==== 
 +The record driver object is responsible for extracting key pieces of data about a record from the data returned by your chosen service.  To create it, follow these steps: 
 +    - Create a [[development:plugins:record_drivers|record driver plugin]] for your data source -- i.e. module/VuFind/src/VuFind/RecordDriver/Sample.php containing a class called \VuFind\RecordDriver\Sample. 
 +    - Decide which existing record driver class to extend.  If the records you are working with are very distinctive, you will probably want to extend \VuFind\RecordDriver\AbstractBase and build custom templates.  However, in many cases, records will contain bibliographic data that is similar enough to VuFind®'s default Solr records to allow you to extend \VuFind\RecordDriver\DefaultRecord (or one of its subclasses) and thus recycle many of VuFind®'s default built-in templates.  This approach is recommended whenever it is possible.
     - Implement key methods:     - Implement key methods:
-      * //Constructor// - The constructor should set the $this->resourceSource property to the source value you chose earlier (i.e. 'Sample') and, if applicable, the $this->recordIni property to an .ini file containing record-related settings (if you want to take advantage of [[building_a_related_record_module|related record]] modules). +      * //setRawData()// - This method is fed data directly from the external service (usually by a class implementing \VuFindSearch\Response\RecordCollectionFactoryInterface and set up by the service factory described above) and is responsible for storing it within the object. This data will be used by all other methods to retrieve specific details. The nature of the data being passed in is determined by your search results class (see below); you can use whatever is most convenient for your situation.  \VuFind\RecordDriver\AbstractBase contains a default implementation of this method which stores the data in the $this->fields property; in many cases, you can use this default, but you may want to modify this method if additional processing is necessary. 
-      * //setRawData()// - This method is fed data directly from the external service and is responsible for storing it within the object. This data will be used by all other methods to retrieve specific details. The nature of the data being passed in is determined by your search results class (see below); you can use whatever is most convenient for your situation.  \VuFind\RecordDriver\AbstractBase contains a default implementation of this method which stores the data in the $this->fields property; in many cases, you can use this default, but you may want to modify this method if additional processing is necessary. +      * //getUniqueId()// - This method returns the record's unique identifier.  All VuFind® records must have a unique identifier of some sort.  Note that this unique ID is combined with the "source" value when VuFind® does lookups, so you do not have to worry about IDs from one service colliding with IDs from another service.
-      * //getUniqueId()// - This method returns the record's unique identifier.  All VuFind records must have a unique identifier of some sort.  Note that this unique ID is combined with the "source" value when VuFind does lookups, so you do not have to worry about IDs from one service colliding with IDs from another service.+
       * //getBreadcrumb()// - This method returns a string used for identifying the record in breadcrumbs, page titles, etc.  Normally it will be a short form title or something similar.       * //getBreadcrumb()// - This method returns a string used for identifying the record in breadcrumbs, page titles, etc.  Normally it will be a short form title or something similar.
-      * ...and the rest - If you are extending \VuFind\RecordDriver\SolrDefault and your saved data is not structured exactly like a Solr response, you should override the public get methods of that class to return information from your chosen format.  (If the data is unavailable for a given method, you don't have to override it -- the base code will handle missing data appropriately).  If you are extending \VuFind\RecordDriver\AbstractBase, you should add public get methods for any data elements you will need to display in your templates.  When possible, try to name your methods consistently with other record drivers (see the [[Record Driver Method Master List]]) since this will improve your chances of being able to reuse certain VuFind components.  +      * ...and the rest - If you are extending \VuFind\RecordDriver\SolrDefault and your saved data is not structured exactly like a Solr response, you should override the public get methods of that class to return information from your chosen format.  (If the data is unavailable for a given method, you don't have to override it -- the base code will handle missing data appropriately).  If you are extending \VuFind\RecordDriver\AbstractBase, you should add public get methods for any data elements you will need to display in your templates.  When possible, try to name your methods consistently with other record drivers (see the [[development:architecture:record_driver_method_master_list|Record Driver Method Master List]]) since this will improve your chances of being able to reuse certain VuFind® components.  
       * In most cases, you may want to adjust other methods dealing with things like tabs to display in record view or supported export formats, but these details can probably wait until after the first phase of development.       * In most cases, you may want to adjust other methods dealing with things like tabs to display in record view or supported export formats, but these details can probably wait until after the first phase of development.
-  **Implement Search Classes** - Create a new directory under module/VuFind/src/VuFind/Search named for the search class ID you picked earlier.  In this new directory, you will create three files:+ 
 +====3. Register Record Driver==== 
 +Add your new record driver to the [[https://github.com/vufind-org/vufind/blob/dev/module/VuFind/src/VuFind/RecordDriver/PluginManager.php|\VuFind\RecordDriver\PluginManager]] (if you are contributing new core code) or by modifying the recorddriver_plugin_manager section of your local module's module.config.php (if you are building this as a local extension); you will often want to set up a factory to pass configurations to the record driver's constructor -- see existing driver configurations (in [[https://github.com/vufind-org/vufind/blob/dev/module/VuFind/src/VuFind/RecordDriver/PluginManager.php|VuFind\RecordDriver\PluginManager]]) for some examples. Be sure that the name you register the driver under matches the name expected by your backend factory (and see the [[development:plugins:record_drivers]] page for some notes on registering custom Solr record drivers). 
 + 
 +====4. Implement Search Classes==== 
 +Create a new directory under module/VuFind/src/VuFind/Search (or in an equivalent path in your local module) named for the search class ID you picked earlier.  In this new directory, you will create three files:
     - //Options.php// - This will contain the \VuFind\Search\Sample\Options class, which must extend \VuFind\Search\Base\Options.  See the existing \VuFind\Search\Solr\Options or \VuFind\Search\WorldCat\Options classes for some examples of how this works.  These are the most important methods to implement in this class:     - //Options.php// - This will contain the \VuFind\Search\Sample\Options class, which must extend \VuFind\Search\Base\Options.  See the existing \VuFind\Search\Solr\Options or \VuFind\Search\WorldCat\Options classes for some examples of how this works.  These are the most important methods to implement in this class:
       * //Constructor// -  The constructor tells the other search classes which .ini files to use for search/facet settings and sets up initial options by reading them in from those .ini files.  (You may at this time wish to establish a Sample.ini file -- you can use config/vufind/Summon.ini and config/vufind/WorldCat.ini for inspiration).       * //Constructor// -  The constructor tells the other search classes which .ini files to use for search/facet settings and sets up initial options by reading them in from those .ini files.  (You may at this time wish to establish a Sample.ini file -- you can use config/vufind/Summon.ini and config/vufind/WorldCat.ini for inspiration).
Line 39: Line 49:
       * //getAdvancedSearchAction()// - This is just like getSearchAction, except it points to the Advanced Search form.  This will normally be something like 'sample-advanced.'  If you do not wish to support advanced searches, simply return false.       * //getAdvancedSearchAction()// - This is just like getSearchAction, except it points to the Advanced Search form.  This will normally be something like 'sample-advanced.'  If you do not wish to support advanced searches, simply return false.
     - //Params.php// - This will contain the \VuFind\Search\Sample\Params class, which must extend \VuFind\Search\Base\Params.  Unless you need to do special parameter processing or add new parameters not supported by the base class, you are not required to implement any methods here -- you can just extend with an empty class.  You'll probably end up adding methods here eventually, but for the initial implementation it is nice to leave this empty -- one less thing to worry about!     - //Params.php// - This will contain the \VuFind\Search\Sample\Params class, which must extend \VuFind\Search\Base\Params.  Unless you need to do special parameter processing or add new parameters not supported by the base class, you are not required to implement any methods here -- you can just extend with an empty class.  You'll probably end up adding methods here eventually, but for the initial implementation it is nice to leave this empty -- one less thing to worry about!
-    - //Results.php// - This will contain the \VuFind\Search\Sample\Results class, which must extend \VuFind\Search\Base\Results.  This is the most important and complex search class, and looking at existing examples like \VuFind\Search\Solr\Results and \VuFind\Search\WorldCat\Results may help you understand it better.  You will have to implement several methods:+    - //Results.php// - This will contain the \VuFind\Search\Sample\Results class, which must extend \VuFind\Search\Base\Results.  This class interacts with your backend to expose results to VuFind®, and looking at existing examples like \VuFind\Search\Solr\Results and \VuFind\Search\WorldCat\Results may help you understand it better.  You will have to implement several methods:
       * //performSearch()// - This method reads the search parameters from the parameters and options objects, uses them to perform a search against your chosen service, and uses the results of the search to populate two object properties: $this->resultTotal, the total number of search results found, and $this->results, an array of record driver objects representing the current page of results requested by the user.       * //performSearch()// - This method reads the search parameters from the parameters and options objects, uses them to perform a search against your chosen service, and uses the results of the search to populate two object properties: $this->resultTotal, the total number of search results found, and $this->results, an array of record driver objects representing the current page of results requested by the user.
       * //getFacetList()// - This method returns facet options related to the current search.  You may need to store extra values in performSearch() to allow the list to be generated.  It is possible that getFacetList will be called before a search has been performed -- in this case, you should call the performAndProcessSearch method to fill in all the details.  (See \VuFind\Search\Solr\Results for an example of this).       * //getFacetList()// - This method returns facet options related to the current search.  You may need to store extra values in performSearch() to allow the list to be generated.  It is possible that getFacetList will be called before a search has been performed -- in this case, you should call the performAndProcessSearch method to fill in all the details.  (See \VuFind\Search\Solr\Results for an example of this).
-      * //getRecord()// - This method takes an ID value and returns a record driver representing the requested record If the record does not existit should throw a \VuFind\Exception\RecordMissing object. + 
-      * //getRecords()// - This static method takes an array of IDs and returns an array of record drivers representing the records It may throw a \VuFind\Exception\RecordMissing object if appropriate.  This method is implemented for you in the \VuFind\Search\Base\Result classso you do not have to write your own -- but if there is an efficient way to request multiple IDs at the same timewriting your own method will improve VuFind'performance; the default base class method simply calls getRecord() repeatedly from a loop+====5Register Search Classes==== 
-  - **Create Controllers** - In order to provide web access to your new code, you will need to create two new controllers in the module/VuFind/src/VuFind/Controller folder:+If you are building code into the project's coreyou should register your new classes in [[https://github.com/vufind-org/vufind/blob/dev/module/VuFind/src/VuFind/Search/Options/PluginManager.php|VuFind\Search\Options\PluginManager]], [[https://github.com/vufind-org/vufind/blob/dev/module/VuFind/src/VuFind/Search/Params/PluginManager.php|VuFind\Search\Params\PluginManager]] and [[https://github.com/vufind-org/vufind/blob/dev/module/VuFind/src/VuFind/Search/Results/PluginManager.php|VuFind\Search\Results\PluginManager]]. However, if are working locally, you will need to register the classes in the ['vufind']['plugin_managers']['search_options']['vufind']['plugin_managers']['search_params'] and/or ['vufind']['plugin_managers']['search_results'] sections of your local module'module.config.php
 + 
 +====6. Create Controllers==== 
 +In order to provide web access to your new code, you will need to create two new controllers in the module/VuFind/src/VuFind/Controller folder (or in an equivalent place in your local module):
     - **Search Controller** - This should have a name like \VuFind\Controller\SampleController and should extend \VuFind\Controller\AbstractSearch.  This controller will handle displaying and processing search forms.  At a bare minimum, it should contain a constructor which sets the $this->searchClassId property to the appropriate value (i.e. 'Sample') and calls the parent constructor.  You can look at the \VuFind\Controller\AbstractSearch class for other available options.  You can also define controller actions if you wish, but this is not necessary -- by default, you will have Home, Results and Advanced actions inherited from the base class.     - **Search Controller** - This should have a name like \VuFind\Controller\SampleController and should extend \VuFind\Controller\AbstractSearch.  This controller will handle displaying and processing search forms.  At a bare minimum, it should contain a constructor which sets the $this->searchClassId property to the appropriate value (i.e. 'Sample') and calls the parent constructor.  You can look at the \VuFind\Controller\AbstractSearch class for other available options.  You can also define controller actions if you wish, but this is not necessary -- by default, you will have Home, Results and Advanced actions inherited from the base class.
-    - **Record Controller** - This should have a name like \VuFind\Controller\SamplerecordController and should extend \VuFind\Controller\AbstractRecord.  This controller will handle displaying records and performing record-related tasks (export, save, etc.).  At a bare minimum, it should contain a constructor which sets the $this->searchClassId property to the appropriate value (i.e. 'Sample') and calls the parent constructor. +    - **Record Controller** - This should have a name like \VuFind\Controller\SamplerecordController and should extend \VuFind\Controller\AbstractRecord.  This controller will handle displaying records and performing record-related tasks (export, save, etc.).  At a bare minimum, it should contain a constructor which sets the $this->sourceId property to the appropriate value (i.e. 'Sample') and calls the parent constructor. 
-  - **Register Controllers** - Now that your controllers are created, you should register them in the invokables array of the controller section of module/VuFind/config/module.config.php.  The entries should look like 'sample' => 'VuFind\Controller\SampleController' and 'samplerecord' => 'VuFind\Controller\SamplerecordController'+ 
-  - **Set Up Routes** - You'll need to edit the module/VuFind/config/module.config.php configuration file to set up all the routes related to your new controllers.  The configuration is designed to make it easy to add new routes:+====7. Register Controllers==== 
 +Now that your controllers are created, you should register them in the factories and aliases arrays of the controller section of module/VuFind/config/module.config.php. In most situations, you can use the standard \VuFind\Controller\AbstractBaseFactory to build both of your controllers, though you can also build a custom factory if you need to inject additional dependencies
 +   
 +====8. Set Up Routes==== 
 +You'll need to edit the module/VuFind/config/module.config.php configuration file to set up all the routes related to your new controllers.  The configuration is designed to make it easy to add new routes:
     - In the $recordRoutes array, add an association between the route name (which should be your source value concatenated with the word "record") and the route's controller name (as registered in controller invokables above).  For example: 'samplerecord' => 'samplerecordcontroller'     - In the $recordRoutes array, add an association between the route name (which should be your source value concatenated with the word "record") and the route's controller name (as registered in controller invokables above).  For example: 'samplerecord' => 'samplerecordcontroller'
     - In the $staticRoutes array, add any necessary search-related routes... for example, 'Sample/Home', 'Sample/Search' and 'Sample/Advanced'.     - In the $staticRoutes array, add any necessary search-related routes... for example, 'Sample/Home', 'Sample/Search' and 'Sample/Advanced'.
-    - Note that, if you are using a [[customizing_vufind_2.0#modules|custom module]], you may need to copy some extra route-building logic from the main VuFind configuration, since these convenience route-building arrays are not present by default in empty modules. +    - Note that, if you are using a [[development:architecture:customizing_vufind#modules|custom module]], you may need to copy some extra route-building logic from the main VuFind® configuration or use a [[development:code_generators|code generator]], since these convenience route-building arrays are not present by default in empty modules 
-  - ** Set Up Templates** - Now that the models and controllers are set up, you need some views.+ 
 +====9. Set Up Templates==== 
 +Now that the models and controllers are set up, you need some views.
     - //Search Templates// - Within your chosen theme, create a templates/sample directory (replacing 'sample' with a lowercased version of your search controller's name).  In this directory, you need to create one template for each action: advanced.phtml, home.phtml and search.phtml.  These templates can simply wrap around the default search templates, overriding a few settings if necessary.  See the existing templates in templates/worldcat for the simplest possible example.     - //Search Templates// - Within your chosen theme, create a templates/sample directory (replacing 'sample' with a lowercased version of your search controller's name).  In this directory, you need to create one template for each action: advanced.phtml, home.phtml and search.phtml.  These templates can simply wrap around the default search templates, overriding a few settings if necessary.  See the existing templates in templates/worldcat for the simplest possible example.
-    - //Record Templates// - If your record driver extends \VuFind\RecordDriver\SolrDefault, you may not need to create any record templates at all -- the defaults should work for you.  However, if you built a custom driver, you will need to create an appropriately-named directory under templates/RecordDriver in your chosen theme.  Look at the existing templates in templates/RecordDriver/SolrDefault to see which filenames you need to create and what sort of content they should contain (note that some are optional and some filenames are dynamically generated based on other record driver options -- i.e. the tab-*.phtml files, which are driven by allowed tab options).+    - //Record Templates// - If your record driver extends \VuFind\RecordDriver\DefaultRecord, you may not need to create any record templates at all -- the defaults should work for you.  However, if you built a custom driver, you will need to create an appropriately-named directory under templates/RecordDriver in your chosen theme.  Look at the existing templates in templates/RecordDriver/DefaultRecord to see which filenames you need to create and what sort of content they should contain (note that some are optional and some filenames are dynamically generated based on other record driver options -- i.e. the tab-*.phtml files, which are driven by allowed tab options).
 ---- struct data ---- ---- struct data ----
 +properties.Page Owner : 
 ---- ----
  
development/howtos/connecting_a_new_external_data_source.1347372627.txt.gz · Last modified: 2014/06/13 13:13 (external edit)