Differences

This shows you the differences between two versions of the page.

--- videos:xml_change_tracking_and_email_alerts [2020/12/24 13:28] – created demiankatz
+++ videos:xml_change_tracking_and_email_alerts [2021/10/22 14:37] – [Video 13: XML Change Tracking and Email Alerts] demiankatz
@@ Line 1: / Line 1: @@
 ====== Video 13: XML Change Tracking and Email Alerts ======
-The thirteenth VuFind instructional video discusses VuFind's email alert / selective dissemination of information (SDI) functionality and builds upon some concepts introduced in the earlier [[videos:oai-pmh_server_and_harvest_functionality|OAI-PMH]] and [[videos:indexing_xml_records|XML Indexing]] videos.
+The thirteenth VuFind® instructional video discusses VuFind®'s email alert / selective dissemination of information (SDI) functionality and builds upon some concepts introduced in the earlier [[videos:oai-pmh_server_and_harvest_functionality|OAI-PMH]] and [[videos:indexing_xml_records|XML Indexing]] videos.
 Video is available as an [[https://vufind.org/video/Email_Alerts.mp4|mp4 download]] or through [[https://www.youtube.com/watch?v=uIIEqyKPNk0&feature=youtu.be|YouTube]].
@@ Line 10: / Line 10: @@
   - [[configuration:email_alerts|Email Alerts Documentation]]
+===== Update Notes =====
+This video was recorded using VuFind 7. In VuFind 8.0, a change was made which impacts the content of this video:
+  * All XML import configuration examples now include an easy mechanism for turning on change tracking; it is no longer necessary to customize the XSLT, but simply to uncomment the relevant line in the matching properties file.
 ===== Transcript =====
 // This is a raw machine-generated transcript; it will be cleaned up as time permits. //
-// Coming soon... //
+ hello in this month's VuFind video
+i'm going to tell you about
+VuFind's email alert feature and
+along the way i'm going to build on some
+concepts that were previously discussed
+in the videos about oai pmh and
+indexing xml records if you haven't
+already watched those videos you might
+want to review them before proceeding
+if you're ready to go forward i'll start
+by pointing out that email alerts are a
+comparatively recent feature
+in VuFind having been introduced in
+release 6.1
+also known as selective dissemination of
+information or
+sdi if you want to be technical about it
+the email alerts
+allow a user to subscribe to a search
+query and receive emails notifying them
+of any new results matching that query
+as they're added to your index
+obviously if we want to be able to
+notify users about new records
+we need to be able to tell which records
+are new
+and this requires us to use VuFinds
+change tracking system
+which we first talked about in the oai
+pmh video where we showed you how to set
+up change tracking for marc records
+since that time uh we've also gone
+through how to index
+xml records and so in this video i'm
+going to show you how to set up change
+tracking for xml records so that you can
+set up email alerts
+regardless of what type of record you're
+working with
+just to very quickly recap the
+point of view finds change tracking
+system
+is to identify at index time uh which
+records are new to the index
+and which records have changed since the
+last time they are indexed
+this uses a couple of solar fields
+called first
+index and last indexed to keep track of
+dates
+and in order to function properly it
+needs to know
+the last change date of each record
+that's being indexed
+fortunately using our oai pmh harvesting
+tools
+it's not too difficult to do this
+so i'm going to go to the terminal
+and switch into my VuFind home
+directory
+so in the xml indexing video as an
+example
+we indexed the expositions open access
+journal hosted at villanova
+and so today i'm going to proceed with
+that example but just make some
+modifications
+to enable change tracking the first
+thing that we need to do
+is as i mentioned we need to know the
+modification dates of records
+so that the change tracking system can
+properly judge what has changed
+and what hasn't when we re-index things
+uh fortunately oai pmh provides date
+information with every record
+it just uh doesn't include it inside the
+record metadata itself it's part of the
+uh oai pmh package that's used to
+deliver
+the metadata but VuFind's
+oai harvester includes a setting called
+inject date which we can use to take
+the modification date out of the oai pmh
+header
+and insert it into the metadata of the
+record so that once it's been harvested
+we can take advantage of it this is
+pretty simple
+i'm just going to edit my local
+harvest oai.ini file which we set up
+in that earlier video and go down to the
+exposition section and
+i'm just going to add the line inject
+date
+equals date stamp so in
+every dublin core record that we harvest
+from
+expositions this will cause the
+harvester
+to add a date stamp tag which will
+contain
+the modification date of that record
+this works very similar
+to the inject id mechanism that we were
+already using to get ids into our
+harvested records
+so now that i've changed my harvesting
+rules i need to actually re-harvest
+all of the records and if i were to
+simply run the harvester now i wouldn't
+get everything
+because oai pmh harvesting is
+incremental
+so every time i run the harvest it will
+only download the records that have been
+added or changed
+since my last harvest however in this
+case because i've actually
+changed the way i want to harvest i
+really want to get everything again
+i don't want just an incremental update
+fortunately it's really easy to force if
+you find to do a full re-harvest
+we just need to delete the directory of
+harvested records and start fresh
+so i'm going to rm minus rf
+local harvest expositions to recursively
+force and delete of the exposition's
+harvest directory
+and now i can repeat my harvest
+by running php harvest oai
+dot php and tell it to harvest the
+expositions repository
+and now in just a few seconds i will
+have my 293 records
+downloaded again and just to show you
+that this really
+worked let me open up
+one of these files
+that we just harvested
+and if i scroll to the end of the first
+line
+i will find that it has added a date
+stamp tag
+with a modification date for this record
+so now that i have harvested data with
+dates in it
+i just need to index it and take
+advantage
+of those dates so that you find can
+track changes
+uh unfortunately vue finds example
+uh xslt sheets for indexing xml
+do not as of this recording
+include modification
+tracking support i have just opened
+a jira ticket VuFind 1461
+which i will link to in the notes
+accompanying this video
+so that a future release if you find
+will include examples to make this
+a little bit more convenient but for
+today
+i'm going to take advantage of this
+shortcoming to
+give an example of customizing an xslt
+and i'll just add the feature to the
+ojs example which we're using for
+harvesting
+expositions so as with
+just about everything else in VuFind
+that you might want to customize
+we can customize an xslt by copying it
+from the core code
+into our local custom directory
+so in this instance the
+existing xslt lives in the local import
+xsl directory uh and so i need to
+create a local equivalent of that so i
+have a place to put my custom version
+so i'm just going to create the
+directory local
+import xsl
+and then i'm going to copy
+from import xsl ojs multirecord
+xsl into
+local import xsl
+since ojs multi-record xsl is the
+style sheet we're using for transforming
+records harvested from ojs in this
+example
+then i just need to
+edit the newly created copy of the xslt
+and i'm just going to go down to the
+bottom
+and put some new fields following this
+url here
+so in xslt we're generating an xml
+document
+in the case of indexing we're creating
+the solar document that adds a record to
+the solar index
+so uh i can just put in
+some solar xml
+field name equals first indexed
+in order to create the first index
+value that we want to store i'm going to
+close that
+field tag and to fill in the actual
+value
+of the first index date VuFind
+provides
+a convenient helper function called get
+first indexed
+which takes a few parameters and uses
+them to calculate the
+first index date so here is where i need
+to do a little bit of xslt
+i use the xsl value of tag
+to tell the xslt processor i want to
+insert a dynamic value here
+and for the select value i'm going to
+use
+the php function call which is how we
+run custom php code from inside an
+xslt the first parameter to this
+is the name of the php function which in
+this case
+is you find get first indexed
+and then we follow this with all of the
+parameters that
+the actual php function requires
+in this case we it takes three first the
+name of the solar core that we are
+indexing into
+which in this case is biblio next
+it takes the id of the record that we
+want to look up
+and this we can get from the identifier
+field
+of our harvested records so i'm just
+going to say
+normalize space string identifier
+which will take the contents of the
+identifier tag
+presented as a string which is the data
+type that
+VuFind expects and normalize all the
+space
+finally the third parameter is the last
+change date of the record that we're
+looking up
+this is the reason we added that gate
+stamp
+tag to our harvested records so again
+i'm going to
+normalize space and convert to string
+the date stamp
+and that is my function fall then i can
+just close
+my value of tag and that should do the
+job
+of course i realize that if you're not
+particularly familiar with xslt
+or php this could be a bit intimidating
+but even if you don't fully understand
+it
+you can copy and paste this out of the
+jira ticket and
+in a future if you find release this can
+all be configured through your
+properties file
+instead of having to do this work now as
+i mentioned there are
+actually two fields we need not just
+first index but also
+last indexed but the code is extremely
+similar for both of them so i'm just
+going to copy and paste
+what i wrote for first indexed
+and change every instance of the word
+first
+to last and that should do the job
+now it all comes down to whether i typed
+that correctly
+let's do a quick test to
+see if i did i'm just going to run
+the xml import
+tool import slash import xsl.php
+in test only mode to look at
+some of my recently harvested records
+um using this
+new
+set of changes
+and if i did it correctly
+after a moment i see
+the generated solar xml
+and sure enough it includes uh
+first index and last index dates showing
+the present time which means it worked
+right because
+all of these records have been indexed
+and VuFind
+as far as it's concerned for the first
+time today
+because this is the first time i've
+indexed these records with change
+tracking turned on
+uh if i were to re-index them in the
+future
+without the records themselves having
+changed at all
+um these dates would stay the same
+that's how VuFind can detect
+what changes similarly if i index new
+copies of the records that have been
+revised and have
+newer modification dates then that last
+index field is going to reflect
+the date of the change
+so having demonstrated in test mode that
+this all works correctly
+now all that's left to do is to actually
+index all these records
+i'm going to do that with the harvest
+batch
+import xsl script i just tell that that
+i want to
+index the expositions directory using
+the ojs.properties configuration
+and in just a few seconds
+all of my records should be stored in
+the index with
+appropriate base
+so that's it for change tracking
+xml we now have everything we need to
+turn
+on the email alert feature and
+fortunately this
+is pretty simple in fact much simpler
+than editing an xslt
+if we look at our local config
+you find config.ini file
+there's an account section which i'm
+going to jump down to
+and inside there there are a few
+settings related to email alerts
+the first is called schedule searches
+which is turned off by default
+until you turn it on all of the
+functionality around email alerts
+is hidden so i'm just going to turn it
+on
+the next setting is called force first
+scheduled email
+the purpose of this is that by default
+if a user subscribes to a search
+they won't get an email until there are
+actually some new records
+added to the search results but if you
+turn this
+on you can change the behavior so that
+the first night after they've
+subscribed they will receive an email
+which sort of proves that the system is
+working
+i'm going to turn that on for today's
+demo uh because otherwise we would have
+to
+uh wait some period of time for things
+to change
+uh before we could see the system
+sending a message
+the final setting here in config.ini
+is scheduled search frequencies which
+allows you to customize
+the frequency at which notifications are
+sent
+by default the user can choose between
+never being notified
+being notified every day if there's a
+change or being notified every week
+if there's a change but if you want to
+have different lengths of time between
+notifications
+you can customize this here but i'm
+going to leave it at the defaults
+for now also
+while i'm in config.ini anyway uh
+i should mention email configuration
+since
+obviously VuFind is only going to be
+able to send notification emails if it
+knows how to send email
+and that may require some additional
+configuration so i just want to bring
+your attention
+to the fact that there's a section in
+config.ini called
+mail which is where you can set up uh
+smtp settings standard mail transfer or
+rather simple mail transfer protocol
+to tell you find how to send mail
+by default it assumes that there is an
+smtp
+process running on the VuFind server
+but if there is not or if you have an
+smtp service that requires
+authentication
+you can adjust these settings to point
+to the right place
+there are a number of services that will
+allow you to
+send emails using smtp in some cases
+without cost if your volume is low or
+you can run your own server
+it's also worth noting that while
+VuFind currently assumes you're using
+smtp
+the laminas mail module that it relies
+on for sending messages
+is quite flexible and can support other
+methods as well
+so with a little bit of custom code
+it's possible to do things like use the
+local send mail program
+it would just require a custom factory
+for your mail service
+and if there's something that you think
+would be useful for the community please
+reach out and suggest it we can always
+add more configuration options
+in the future the other
+possibility here is that you can set
+your mail to test only
+which will cause it to pretend to send
+emails but not
+actually send them obviously in a real
+world situation you never want to do
+that
+but for this demo since i don't have an
+smtp server
+to show you right now i'm just going to
+put this in test mode
+so it will appear to be working even
+though nothing will really be happening
+so that's it we've configured email
+alerts
+so now let me show you what this looks
+like
+in the user interface
+so i'm going to go to my catalog i'm
+going to search for
+test and having performed this search
+it's now in my search history so if i
+scroll down to search options and
+go to the search history screen you'll
+see that
+there's an alert scheduled drop down
+that did not used to be here
+this is how users are able to
+uh get alerts so i'm going to set myself
+up for daily alerts i want to know if
+there are new tests
+and of course i have to log in because
+it needs to know my email address which
+requires me to have an account so it
+prompts me to do that
+i log in and now as you can see
+the search has moved from the recent
+searches list into my saved searches
+list
+and it's scheduled for daily alerts
+i could easily change this to
+unsubscribe or improve the
+increase the frequency if i wanted to
+but for now i'll just leave it here
+so how do the alerts get sent simple
+there is a command line tool
+which does the job and i will run that
+right now
+with php public index.php
+scheduled search slash notify
+and as you can see it was processing one
+saved search
+found my search because
+i um had that setting to
+always send an email the first time
+someone subscribes
+it treats this as new so it sent a
+virtual message
+and now it's done so
+if you want to use this feature all you
+need to do
+is add the scheduled search notify
+command to a cron job that runs
+probably on a daily basis and then your
+server will send out emails as things
+change
+if you already have a script set up to
+uh index records on a regular basis
+popping in this notification at the end
+of that script would make a lot of sense
+to ensure that the latest information
+gets sent out
+at the moment that it is fresh and
+that's
+all there is to email alerts uh thanks
+for watching and as always feel free to
+reach out with questions or suggestions
+see you in the next video
 ---- struct data ----
 ----